Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping 101 with Python

gregreda.com

Post reply on HN
Print view View original

71–80 of 80 posts

Re: Web Scraping 101 with Python

#71

Post by steeve »

pmorici wrote: post #19

BeautifulSoup, has received a lot of positive press on HN over the years so when I needed to do some heavy scraping I gave it a spin. It was a total disappointment. It's fine if you are scraping a small set of similar pages from a single site but if you are scraping a large number of pages across many sites and esp pages with text encoding other than ascii / utf-8 it choaks so frequently as to be useless. BeautifulSo…

lxml.HTML (+ html5lib if needed) is a FAR superior choice indeed. It's an order of magnitude faster, and you can use not only xpath selectors, but also CSS selectors too, from a lxml dom. or indeed you can use Scrapy, which fixes all these encoding BS, handles per domain crawl rate and concurrent requests etc...

Re: Web Scraping 101 with Python

#72

Post by cuppster »

samarudge wrote: post #6

PyQuery is pretty awesome ( https://pypi.python.org/pypi/pyquery ) Using Requests to download the document, pump it into PyQuery and you can use any jQuery style selectors to get text, attributes and all sorts of other stuff. Example; Here's how to scrape the hacker news homepage https://gist.github.com/samarudge/035ab8aaca224415cb49 (that code could probably be improved but I only spent a couple of minutes on it)

Watch out for unicode when using pyquery and requests. I provided a fix for that just recently now merged into the pyquery repo. I use it (among other things) to scrape upcoming comic book releases =) http://cuppster.com/2013/01/30/decorators-scrapers-and-gener...

Re: Web Scraping 101 with Python

#74

Post by josephagoss »

josephagoss wrote: post #73

What would you recommend for automating website interaction (use a bot to get betting numbers and then log in and automate a bet without any human interaction) some sites use an API (betfair) but some don't.

And those numbers are being posted using ajax or something else (updating in real time)

Re: Web Scraping 101 with Python

#76

Post by thibauts »

danneu wrote: post #63
thibauts wrote: post #56

Earlier quoted context omitted.

I'll second, lxml.html is in my experience very robust and fast. I've been writing a lot of scrapers along the years and the best combination I found so far is requests / lxml.html / gevent. It doesn't get any simpler than this IMO http://pastebin.com/hacxmAjV

For fun here's the Ruby+Nokogiri version and my attempt at the Clojure+Enlive version (3rd day learning Clojure). https://gist.github.com/danneu/5131596

Pretty nice. I'd like to see how it goes in a real world scenario with concurrency, manipulation of extracted nodes and general HTTP post/auth/etc. I'm not sure about Ruby+Nokogiri but Clojure+Enlive may well be a great choice.

Re: Web Scraping 101 with Python

#77

Post by Riesling »

samarudge wrote: post #6

PyQuery is pretty awesome ( https://pypi.python.org/pypi/pyquery ) Using Requests to download the document, pump it into PyQuery and you can use any jQuery style selectors to get text, attributes and all sorts of other stuff. Example; Here's how to scrape the hacker news homepage https://gist.github.com/samarudge/035ab8aaca224415cb49 (that code could probably be improved but I only spent a couple of minutes on it)

I also prefer PyQuery over Beautiful Soup.

Especially since you can use a Chrome or FFX plugin to inject jQuery into any webpage and then refine your selector via the JavaScript console.

All you need to do then is to copy the selector in your python script and you are done.

Re: Web Scraping 101 with Python

#79

Post by shane42 »

bjourne wrote: post #34
niggler wrote: post #3

For the pythonistas: what is the relationship between BeautifulSoup, lxml, urllib*, scrapy and mechanize?

Here are my highly opinionated opinions about their respective use cases: * BeautifulSoup: It was the best scraping library ever until python-lxml came around and stole the show. Despite that the manual said BeautifulSoup gives you unicode, damnit! it had some long-standing bugs which it gave you strings or incorrectly decoded web pages. I wouldn't use it anymore because lxml is strictly superior. * lxml: The king of…

I would look at "delta-scraping" as one of the things that is easier with scrapy. There is an existing extension for it: https://github.com/scrapinghub/scrapylib/blob/master/scrapyl... and lots of support and help from the community.

Admittedly writing this from scratch requires learning the framework, but being able to share and reuse code is a huge win (disclaimer: I'm a Scrapy contributor).

Re: Web Scraping 101 with Python

#80

Post by johnnyg »

ankimal wrote: post #53
johnnyg wrote: post #39

Earlier quoted context omitted.

Mechanize is far and away the best and easiest way to scrape with Ruby until anything is rendered in javascript, which is explicitly not supported. I tend to use Mechanized until I can't, then switch to Watir. Over time, I've found myself just strait up picking up Watir as it runs your browser directly and supports javascript rendering as a result.

How is performance with Watir? With casperjs a page takes me on an avg. 5-10 secs. to process.

Not great. About the same...
Post reply on HN