python - Encoding Emojis with Beautiful Soup -


looking help. working on project scraping specific craigslist posts using beautiful soup in python. can display emojis found within post title have been unsuccessful within post body. i've tried different variations nothing has worked far. appreciated.

code:

f = open("clcondensed.txt", "w") html2 = requests.get("https://raleigh.craigslist.org/wan/6078682335.html") soup = beautifulsoup(html2.content,"html.parser") #post title  title = soup.find(id="titletextonly")        title1 = soup.title.string.encode("ascii","xmlcharrefreplace") f.write(title1) #post body   body = soup.find(id="postingbody")           body = str(body) body = body.encode("ascii","xmlcharrefreplace") f.write(body)

error received body:

'ascii' codec can't decode byte 0xef in position 273: ordinal not in range(128)

you should use unicode

body = unicode(body)

please refer beautiful soup documentation navigablestring

update:

sorry quick answer. it's not right.

here should use lxml parser instead of html parser, because html parser not support ncr (numeric character reference) emoji.

in test, when ncr emoji decimal value greater 65535, such html demo emoji 🚢, html parser decode wrong unicode \ufffd u"\u0001f6a2". can not find accurate beautiful soup reference this, lxml parser ok.

below tested code:

import requests bs4 import beautifulsoup f = open("clcondensed.txt", "w") html = requests.get("https://raleigh.craigslist.org/wan/6078682335.html") soup = beautifulsoup(html.content, "lxml") #post title title = soup.find(id="titletextonly") title = unicode(title) f.write(title.encode('utf-8')) #post body body = soup.find(id="postingbody") body = unicode(body) f.write(body.encode('utf-8')) f.close()

you can ref lxml entity handling more things.

if not install lxml, ref lxml installing.

hope help.


Comments

Post a Comment

Popular posts from this blog

Retrieving ETA (estimated time of arrival) with Google Distance Matrix API and public transit as transport mode -

Image
is possible retrieve eta google's distancematrix api (or alternative api). fyi, eta different duration, when using public transport. following example clarify request. when leave on friday @ 8.15 i'll able catch train @ 8.30 , arrive @ 9.00 am. google says duration 30 minutes. however, when leave on saturday @ 8.15 i'll able catch first train @ 9.00 , arrive @ 9.30 am. google still says duration 30 minutes, means ignore delay. google map supports functionality there api around giving me eta instead of duration. can see, distance-matrix' response no good. clarifying question little. eta estimated time of arrival , it's clear user wants know entire duration of transport (including waiting time public transport).
Read more

android - ConstraintLayout: Realign baseline constraint in case if dependent view visibility was set to GONE -

Image
is there way realign baseline constraint textview in case if dependent view visibility set gone? my layout code: <android.support.constraint.constraintlayout android:layout_width="match_parent" android:layout_height="wrap_content"> <textview android:id="@+id/title" android:layout_width="wrap_content" android:layout_height="wrap_content" android:text="test title" android:textcolor="@color/black" android:textsize="24sp" app:layout_constraintleft_toleftof="parent" app:layout_constraintright_torightof="parent"/> <textview android:id="@+id/subtitle1" android:layout_width="wrap_content" android:layout_height="wrap_content" android:text="subtitle 1" android:textsize="11sp" app:layout_constraintb...
Read more

c# - Populating Gridview inside Listview ItemTemplate On Web User Control from Code Behind -

i have think simple question, might more complex.. i have spent few days looking answer on google , various questions on site cannot seem come right. what trying bind gridview on user control ascx page default.aspx.cs on page load event. user control markup follows: <%@ control language="c#" autoeventwireup="true" codebehind="vulnerabilityexternalip.ascx.cs" inherits="vulnerabilityassesment.controls.vulnerabilityexternalip" %> <asp:table runat="server" cellpadding="1" cellspacing="2"> <asp:tablerow> <asp:tablecell> <asp:label id="lblvulnerabilityname" runat="server" text="vulnerability name:" cssclass="itemname"></asp:label> </asp:tablecell> <asp:tablecell> <asp:label id="lblvulnerabilitynametext" runat="server" text='<%# eval("mainvulnerabilityname")...
Read more