Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping 101 with Python

gregreda.com

Post reply on HN
Print view View original

41–50 of 80 posts

Re: Web Scraping 101 with Python

#41

Post by SeppoErviala »

bjourne wrote: post #34
niggler wrote: post #3

For the pythonistas: what is the relationship between BeautifulSoup, lxml, urllib*, scrapy and mechanize?

Here are my highly opinionated opinions about their respective use cases: * BeautifulSoup: It was the best scraping library ever until python-lxml came around and stole the show. Despite that the manual said BeautifulSoup gives you unicode, damnit! it had some long-standing bugs which it gave you strings or incorrectly decoded web pages. I wouldn't use it anymore because lxml is strictly superior. * lxml: The king of…

There's even lxml.cssselect if you prefer css selectors over Xpaht.

Re: Web Scraping 101 with Python

#42

Post by ville »

tyilo wrote: post #17

Why is he using lxml as the parser and not just the one built into BS?

BeautifulSoup uses regular expressions! http://stackoverflow.com/questions/1732348/regex-match-open-...

Like many here point out, lxml is a fast and versatile library that could be used for this alone without BS. lxml.html can parse HTML and lxml also has support for using HTML5 parser from html5lib that deals with broken HTML in the standardized way.

Re: Web Scraping 101 with Python

#43

Post by ville »

chewxy wrote: post #31
kimagure wrote: post #20

Earlier quoted context omitted.

PyQuery seems to always be faster in my experience than BS4 (for ripping the same information). Anyone else have a similar experience?

Only on wellformed pages. There are many many many many many malformed pages on the internet. Even those that are created in 2013

Fortunately HTML5 defined a standard way to parse even broken HTML and that parser is implemented in html5lib package. You can use it also with lxml and even use "jQuery like" selectors with lxml.cssselect (http://lxml.de/cssselect.html)

Re: Web Scraping 101 with Python

#45

Post by Encosia »

sergiotapia wrote: post #14

Here are some awesome libraries I've used for HTML scraping: 1. Python - BeautifulSoup 2. Ruby - Nokogiri (use in conjunction with Watir if you're scraping a very client-heavy website). 3. C# - HtmlAgilityPack in conjunction with ScrapySharp (there's a nuget package for both) - I highly recommend ScrapySharp because it allows you to query elements using a very familiar Css selector type similar to how you query dom e…

CsQuery is another interesting library for C# along those same lines: https://github.com/jamietre/CsQuery

Re: Web Scraping 101 with Python

#46

Post by d0mine »

wslh wrote: post #26
pmorici wrote: post #19

BeautifulSoup, has received a lot of positive press on HN over the years so when I needed to do some heavy scraping I gave it a spin. It was a total disappointment. It's fine if you are scraping a small set of similar pages from a single site but if you are scraping a large number of pages across many sites and esp pages with text encoding other than ascii / utf-8 it choaks so frequently as to be useless. BeautifulSo…

In my experience lxml.html is much better.

The article shows that BeautifulSoup can use lxml internally. It can also use html5lib.

Re: Web Scraping 101 with Python

#47

Post by zevyoura »

boyter wrote: post #9
niels wrote: post #7

This will only be a good approach if you are going to scrape a small amount of pages. The problem is using synchronous requests, as this blocks the crawler until a request has finished. Using asynchronous requests such as supported by twisted (and scrapy) will allow you to crawl a lot faster using the same resources.

This can actually sometimes be a feature. It makes it far less likely to have your IP banned. Its also a far more polite way to crawl someones site.

I would argue that the proper implementation provides real rate limiting, both in terms of max requests per second and also max concurrent requests. Limiting to one concurrent request is likely to be extremely slow for any significant amount of data, and a couple concurrent requests is not impolite. Obviously I'm not saying you should effectively DoS the site you're scraping, but there's a balance and 1 concurrent request is almost definitely the wrong place to set it.

Re: Web Scraping 101 with Python

#48

Post by rscale »

ankimal wrote: post #33
sergiotapia wrote: post #14

Here are some awesome libraries I've used for HTML scraping: 1. Python - BeautifulSoup 2. Ruby - Nokogiri (use in conjunction with Watir if you're scraping a very client-heavy website). 3. C# - HtmlAgilityPack in conjunction with ScrapySharp (there's a nuget package for both) - I highly recommend ScrapySharp because it allows you to query elements using a very familiar Css selector type similar to how you query dom e…

Also recommend Mechanize for Ruby (uses nokogiri under the covers). http://mechanize.rubyforge.org/ http://phantomjs.org/ or "its easier to deal with cousin" http://casperjs.org/ for very client heavy sites. FWIW, you can get away with HTML only scrapers most of the time, you just need to look harder to find all the data. Totally recommend using "View page source" as that would always give you the original HTML vs th…

I recommend Selenium before I'd recommend PhantomJS in situations where Mechanize/Nokogiri don't cut the mustard,

I've found Selenium scripts much easier to comprehend, modify, and maintain over time than the PhantomJS scripts.

Re: Web Scraping 101 with Python

#49

Post by zevyoura »

dpearson wrote: post #35
sergiotapia wrote: post #14

Here are some awesome libraries I've used for HTML scraping: 1. Python - BeautifulSoup 2. Ruby - Nokogiri (use in conjunction with Watir if you're scraping a very client-heavy website). 3. C# - HtmlAgilityPack in conjunction with ScrapySharp (there's a nuget package for both) - I highly recommend ScrapySharp because it allows you to query elements using a very familiar Css selector type similar to how you query dom e…

> Scraping online content is so simple these days, if a website doesn't offer an API you still have alternatives ;) Having written code to both leverage a site's (private) API and to scrape that same site (when the private API stopped existing), I would much, much rather use an API. Yes, the scraper works, but the scraper's code is much messier, and JSON keys, for example, provide some documentation in their own righ…

Another point in favor of APIs is that scrapers are very brittle, and likely to break with changes to content presentation. Also, scrapers have a lot more overhead as you need to both receive and parse the markup data.

Re: Web Scraping 101 with Python

#50

Post by d0mine »

pmorici wrote: post #19

BeautifulSoup, has received a lot of positive press on HN over the years so when I needed to do some heavy scraping I gave it a spin. It was a total disappointment. It's fine if you are scraping a small set of similar pages from a single site but if you are scraping a large number of pages across many sites and esp pages with text encoding other than ascii / utf-8 it choaks so frequently as to be useless. BeautifulSo…

> with text encoding other than ascii / utf-8 it choaks so frequently as to be useless

I had the opposite experience though I used it only "for small jobs".

If there are no Content-Type (either in http headers or meta http-equiv) that specifies character encoding, no meta charset, and no xml declaration for xhtml, etc that is if the only way to find out character encoding is to guess then even in this case BeautifulSoup includes UnicodeDammit that uses chardet to guess the encoding.

Post reply on HN