21–30 of 80 posts
1. It's a headless WebKit browser, so it plays well with javascript and (sorta) flash. 2. It's easy to capture screenshots of the pages you're scraping, which is great for sanity checks later on.
Http://phantomjs.org - the main engine Http://Casperjs.org - syntactic sugar for phantomjs
1: http://jsoup.org/ 2: http://fluffyelephant.com/2012/09/crawling-orbitz-com/
But why python? Stop now and use perl. Perl's HTTP libraries are actually sane, unlike urllib[2], WWW::Mechanize is brilliant, and it's easier to throw in disgusting hacks in perl when you need them, which is constantly in the business of scraping the web.
If you're coming from a primarily Python background you won't just be waltzing right in and using CPAN modules right away. You have to understand the underlying language as well. Python and perl's object oriented systems for example are quite different (using Moose helps somewhat granted the person knows it even exists). Then there's the issue of understanding various contexts. These differences may take time getting used to depending on the level of the developer.
Python also supports regex as well, which can be useful for weird situations. Granted however it's not going to be as tightly integrated as perl. There's even a Natural Language Toolkit if you want to get really crazy with things.
TLDR: Right tool for the job should take environmental circumstances into account as well
But why python? Stop now and use perl. Perl's HTTP libraries are actually sane, unlike urllib[2], WWW::Mechanize is brilliant, and it's easier to throw in disgusting hacks in perl when you need them, which is constantly in the business of scraping the web.
BeautifulSoup, has received a lot of positive press on HN over the years so when I needed to do some heavy scraping I gave it a spin. It was a total disappointment. It's fine if you are scraping a small set of similar pages from a single site but if you are scraping a large number of pages across many sites and esp pages with text encoding other than ascii / utf-8 it choaks so frequently as to be useless. BeautifulSo…
BeautifulSoup, has received a lot of positive press on HN over the years so when I needed to do some heavy scraping I gave it a spin. It was a total disappointment. It's fine if you are scraping a small set of similar pages from a single site but if you are scraping a large number of pages across many sites and esp pages with text encoding other than ascii / utf-8 it choaks so frequently as to be useless. BeautifulSo…
This will only be a good approach if you are going to scrape a small amount of pages. The problem is using synchronous requests, as this blocks the crawler until a request has finished. Using asynchronous requests such as supported by twisted (and scrapy) will allow you to crawl a lot faster using the same resources.
BeautifulSoup, has received a lot of positive press on HN over the years so when I needed to do some heavy scraping I gave it a spin. It was a total disappointment. It's fine if you are scraping a small set of similar pages from a single site but if you are scraping a large number of pages across many sites and esp pages with text encoding other than ascii / utf-8 it choaks so frequently as to be useless. BeautifulSo…
what would you suggest?
But why python? Stop now and use perl. Perl's HTTP libraries are actually sane, unlike urllib[2], WWW::Mechanize is brilliant, and it's easier to throw in disgusting hacks in perl when you need them, which is constantly in the business of scraping the web.
There is a port of mechanize to Python as well, if that is your main reason to use Perl.
Whatever you can do in "disgusting hacks" can be done just as quickly in a way which won't make you want to vomit when you look over the code again later.