python - UnicodeDecodeError: 'ascii' codec can't decode byte in Textranking code -


this question has answer here:

when execute below code

import networkx nx import numpy np nltk.tokenize.punkt import punktsentencetokenizer sklearn.feature_extraction.text import tfidftransformer, countvectorizer  def textrank(document):     sentence_tokenizer = punktsentencetokenizer()     sentences = sentence_tokenizer.tokenize(document)      bow_matrix = countvectorizer().fit_transform(sentences)     normalized = tfidftransformer().fit_transform(bow_matrix)      similarity_graph = normalized * normalized.t      nx_graph = nx.from_scipy_sparse_matrix(similarity_graph)     scores = nx.pagerank(nx_graph)     return sorted(((scores[i],s) i,s in enumerate(sentences)), reverse=true)  fp = open("qc")     txt = fp.read() sents = textrank(txt) print sents

i following error

traceback (most recent call last):   file "textrank.py", line 44, in <module>     sents = textrank(txt)   file "textrank.py", line 10, in textrank     sentences = sentence_tokenizer.tokenize(document)   file "/usr/local/lib/python2.7/dist-packages/nltk/tokenize/punkt.py", line 1237, in tokenize     return list(self.sentences_from_text(text, realign_boundaries))   file "/usr/local/lib/python2.7/dist-packages/nltk/tokenize/punkt.py", line 1285, in sentences_from_text     return [text[s:e] s, e in self.span_tokenize(text, realign_boundaries)]   file "/usr/local/lib/python2.7/dist-packages/nltk/tokenize/punkt.py", line 1276, in span_tokenize     return [(sl.start, sl.stop) sl in slices]   file "/usr/local/lib/python2.7/dist-packages/nltk/tokenize/punkt.py", line 1316, in _realign_boundaries     sl1, sl2 in _pair_iter(slices):   file "/usr/local/lib/python2.7/dist-packages/nltk/tokenize/punkt.py", line 311, in _pair_iter     el in it:   file "/usr/local/lib/python2.7/dist-packages/nltk/tokenize/punkt.py", line 1291, in _slices_from_text     if self.text_contains_sentbreak(context):   file "/usr/local/lib/python2.7/dist-packages/nltk/tokenize/punkt.py", line 1337, in text_contains_sentbreak     t in self._annotate_tokens(self._tokenize_words(text)):   file "/usr/local/lib/python2.7/dist-packages/nltk/tokenize/punkt.py", line 1472, in _annotate_second_pass     t1, t2 in _pair_iter(tokens):   file "/usr/local/lib/python2.7/dist-packages/nltk/tokenize/punkt.py", line 310, in _pair_iter     prev = next(it)   file "/usr/local/lib/python2.7/dist-packages/nltk/tokenize/punkt.py", line 577, in _annotate_first_pass     aug_tok in tokens:   file "/usr/local/lib/python2.7/dist-packages/nltk/tokenize/punkt.py", line 542, in _tokenize_words     line in plaintext.split('\n'): unicodedecodeerror: 'ascii' codec can't decode byte 0xe2 in position 9: ordinal not in range(128)

i executing code in ubuntu. text, referred website https://uwaterloo.ca/institute-for-quantum-computing/quantum-computing-101. created file qc (not qc.txt) , copy pasted data paragraph paragraph file. kindly me resolve error. thank you

please try if following works you. 

import networkx nx import numpy np import sys  reload(sys) sys.setdefaultencoding('utf8')  nltk.tokenize.punkt import punktsentencetokenizer sklearn.feature_extraction.text import tfidftransformer, countvectorizer  def textrank(document):     sentence_tokenizer = punktsentencetokenizer()     sentences = sentence_tokenizer.tokenize(document)      bow_matrix = countvectorizer().fit_transform(sentences)     normalized = tfidftransformer().fit_transform(bow_matrix)      similarity_graph = normalized * normalized.t      nx_graph = nx.from_scipy_sparse_matrix(similarity_graph)     scores = nx.pagerank(nx_graph)     return sorted(((scores[i],s) i,s in enumerate(sentences)), reverse=true)  fp = open("qc")     txt = fp.read() sents = textrank(txt.encode('utf-8')) print sents

Comments

Post a Comment

Popular posts from this blog

Command prompt result in label. Python 2.7 -

i want put result(output) of shell command label. command works, label shows "0", however, in command prompt, result shown correctly, need showed in label. doing ubuntu. myg1 = button(root, text="rodyti informacija", command=lambda: gauti()) myg1.pack(side=bottom) def gauti(): imti = tekstas.get("1.0", "end-1c") info = subprocess.call("id '{imti}' ".format(imti=imti), shell=true) w = label(root, text= info) w.pack(side = bottom) subprocess.call() returns exit code of process created (in case 0 success). if want text output of process, should instead call subprocess.check_output() .
Read more

javascript - How do I use URL parameters to change link href on page? -

i change href value of link using url parameters. example: default <a id="dlink" href="link-1">link</a> but, when user goes https://url.com/?dlink=link-2 it swaps out link new link <a id="dlink" href="link-2">link</a> first need value url. if you're not using javascript framework can create function using jquery. please check get url parameter jquery or how query string values in js it. after that, element "a" id , set value there. $("#dlink").attr("href", hreffromparam);
Read more

amazon web services - AWS Route53 Trying To Get Site To Resolve To www -

so have domain nameservers pointed aws route 53. site hosted on ec2 instance , reachable resolves site.com, if input www.site.com. i'm curious how site resolve www.site.com. have under route53 domains hosted zone 2 additional record sets outside of ns , soa: a site.com ec2 elastic ip cname www.site.com ec2 public dns i can reach site using both site.com , www.site.com, resolve site.com. there way make if enter either, resolves www.site.com? at first, both should record set. a site.com ec2 elastic ip www.site.com ec2 elastic ip and need redirect setting in web server (nginx, apache, ..) following. (source: how redirect www non-www nginx on centos 7 ) redirect non-www www if want redirect users plain, non-www domain www domain, add server block: server { server_name example.com; return 301 $scheme://www.example.com$request_uri; } save , exit. configures nginx redirect requests "example.com" "www.example.com". not
Read more