Web Scraping 101 with Python
gregreda.com
Web Scraping 101 with Python
1–10 of 80 posts
Re: Web Scraping 101 with Python
#2For the Rubyists out there, I was just writing a crawler this morning, and I really like Nokogiri ( http://nokogiri.org/ ).
Re: Web Scraping 101 with Python
#3For the pythonistas: what is the relationship between BeautifulSoup, lxml, urllib*, scrapy and mechanize?
Re: Web Scraping 101 with Python
#4Using Requests and lxml is a better solution — except when you need many concurrent spiders, in which case you should be using Scrapy (you'd probably be looking for the Middleware discussed here, too: https://groups.google.com/d/msg/scrapy-users/WqMLnKbA43I/B3N...).
Re: Web Scraping 101 with Python
#5But why python? Stop now and use perl. Perl's HTTP libraries are actually sane, unlike urllib[2], WWW::Mechanize is brilliant, and it's easier to throw in disgusting hacks in perl when you need them, which is constantly in the business of scraping the web.
Re: Web Scraping 101 with Python
#6PyQuery is pretty awesome (https://pypi.python.org/pypi/pyquery)
Using Requests to download the document, pump it into PyQuery and you can use any jQuery style selectors to get text, attributes and all sorts of other stuff.
Example; Here's how to scrape the hacker news homepage https://gist.github.com/samarudge/035ab8aaca224415cb49 (that code could probably be improved but I only spent a couple of minutes on it)
Re: Web Scraping 101 with Python
#7This will only be a good approach if you are going to scrape a small amount of pages. The problem is using synchronous requests, as this blocks the crawler until a request has finished. Using asynchronous requests such as supported by twisted (and scrapy) will allow you to crawl a lot faster using the same resources.
Re: Web Scraping 101 with Python
#8But why python? Stop now and use perl. Perl's HTTP libraries are actually sane, unlike urllib[2], WWW::Mechanize is brilliant, and it's easier to throw in disgusting hacks in perl when you need them, which is constantly in the business of scraping the web.
There are other python libraries than urllib.
Re: Web Scraping 101 with Python
#9This will only be a good approach if you are going to scrape a small amount of pages. The problem is using synchronous requests, as this blocks the crawler until a request has finished. Using asynchronous requests such as supported by twisted (and scrapy) will allow you to crawl a lot faster using the same resources.
This can actually sometimes be a feature. It makes it far less likely to have your IP banned. Its also a far more polite way to crawl someones site.
Re: Web Scraping 101 with Python
#10This will only be a good approach if you are going to scrape a small amount of pages. The problem is using synchronous requests, as this blocks the crawler until a request has finished. Using asynchronous requests such as supported by twisted (and scrapy) will allow you to crawl a lot faster using the same resources.
This can actually sometimes be a feature. It makes it far less likely to have your IP banned. Its also a far more polite way to crawl someones site.
I agree, and for a 101 web scraping tutorial keeping it simple is nice.