I have done quite a fair bit of scraping over the last year, and I have to say that the combo of PhantomJS / CasperJS is really unbeatable. I have had to navigate some fairly awful DOM structures replete with errors, confirm dialogs, IE-only features, horrendous endless iframe trees, and more fun stuff. There's nothing I haven't been able to plow through yet using Phantom/Casper.
Web Scraping 101 with Python
61–70 of 80 posts
Re: Web Scraping 101 with Python
#62I'm not generally a huge fan of javascript, but phantomjs/casperjs are far and away the best tools I've used for scraping. Two features that stood out: 1. It's a headless WebKit browser, so it plays well with javascript and (sorta) flash. 2. It's easy to capture screenshots of the pages you're scraping, which is great for sanity checks later on. Http://phantomjs.org - the main engine Http://Casperjs.org - syntactic s…
They make scraping as easy as finding the right jquery selectors (once you inject jQuery onto the page) but can be very slow as compared to a vanilla HTML only scraper. In my experience, a phantom/casper implementation could take upwards of 5-10 secs. to process a single page (almost 5-10x slower). This, even if you disable load of remote images and plugins.
There is a startup penalty to getting phantomjs executable up (including all of its WebKit internals), but once you're there, I've never had any performance issues. Roll a script using casper.each() and feed it an array of urls. It is typically very fast for me. You can trap on the page loaded event and do some benchmarking, but I would disagree with your premise that using PhantomJS/CasperJS is slow.
Re: Web Scraping 101 with Python
#63Earlier quoted context omitted.
In my experience lxml.html is much better.
I'll second, lxml.html is in my experience very robust and fast. I've been writing a lot of scrapers along the years and the best combination I found so far is requests / lxml.html / gevent. It doesn't get any simpler than this IMO http://pastebin.com/hacxmAjV
For fun here's the Ruby+Nokogiri version and my attempt at the Clojure+Enlive version (3rd day learning Clojure).
Re: Web Scraping 101 with Python
#64I'm surprised this article didn't mention scrapy. I had to do a lot of web scraping for a healthcare-related project last month and found scrapy incredibly fast and easy to use.
I've been using BeautifulSoup for a couple years, so it's what I'm most comfortable with. I'd heard of scrapy before, but had never given it a seriously look. That'll change based on all the positive things I've read about it in this thread.
Re: Web Scraping 101 with Python
#65Not to rain on the parade of this post (I'm in support of more people learning to scrape, and more services out there giving us easier to access data). I'm someone who loves web scraping, but I'm also someone who believes that if you don't know what the library is doing, you shouldn't be using it. You can give a brief overview of how to use it and what to look for in the page to extract from, but you're giving a very…
I'm definitely not advocating for people not understanding the problem they want solved. That said, your post sounds empty. Can you elaborate on why your own scrapers that you write from scratch make it all better? How do you your scrapers deal with encode detection, broken html, content prioritization and so forth? I don't like the current options we've got in pythonland, but just writing: "this sucks, so I write my…
Sorry if it sounded empty, there is a reason why I didn't include examples. I'm not really saying "don't use libraries", more just that you should understand the problem first before looking for an easy solution. To be honest, I've done all my scraping in PHP/Perl over the years. Only recently have I started to look into other options such as Python and NodeJS (hence looking at this thread).
I don't claim that my scrapers are better off because they are written from scratch, but they do the job that I want them to do. If I find a target that has a "quirk" I write that into my classes to be used then and in later instances. The real point of doing it this way is more about knowing what the scrapper is doing, rather than what it might do. When you're scraping, you're walking a fine line. Targets may be fine with you doing it to them, but as soon as your scraper freaks out then starts hammering the site, you're in trouble (even worse if you end up doing damage to the target).
I'm not saying that 3rd party libraries are prone to doing this, more so if you forget to set an option or handle an exception, you might screw yourself. If you wrote the scraper it's your own fault for not handling the issue properly. If you used a 3rd party library and the library bugged out causing the issue, you can't really go after the writers, right?
This all comes back to understanding your target, and to understand them, you need some form of knowledge on how it all works.
In response to your questions - I do a lot of things manually when setting up the scrapers. I don't import the data into any sort of DOM (due to watching memory), and in doing that I'm not really concerned about Encoding (for the record I'm generally dealing with UTF-8 and Shift_JIS only) or Broken HTML (I do a general check over the source to see if the layout has changed. If it has, it exits gracefully sending me update notifications on what changed, then puts itself out of action until I reset it. If it's a mission critical scraper, lets just say that I have a myriad of alerts that are sent to me). It's probably not the best way of doing things but it works for me.
Sorry if I was vague, I probably should have put some sort of rant-detection on my mouth. If I didn't answer something specifically, it's not that I was ignoring it, it probably just fell into the "I don't trust it so I don't use it" category. Again, not advocating that people shouldn't use 3rd party libraries, just that you should at least know what you are doing before you do.
Re: Web Scraping 101 with Python
#66I use http://scraperwiki.com It's pretty neat and handy! For beginners here's a tutorial I wrote sometime back http://blog.sanspace.in/scraperwiki/
Re: Web Scraping 101 with Python
#67lxml.html is great for most cases.
I developed this one for my own web scraping: http://docs.webscraping.com/
Re: Web Scraping 101 with Python
#68BeautifulSoup, has received a lot of positive press on HN over the years so when I needed to do some heavy scraping I gave it a spin. It was a total disappointment. It's fine if you are scraping a small set of similar pages from a single site but if you are scraping a large number of pages across many sites and esp pages with text encoding other than ascii / utf-8 it choaks so frequently as to be useless. BeautifulSo…
what would you suggest?
Why not both?
http://www.crummy.com/software/BeautifulSoup/bs4/doc/#specif...
waiter there's some lxml in my soup!
Re: Web Scraping 101 with Python
#69I'm working on a project right now with PHP's built in functions and the help of Google Chrome developer's tools Copy Xpath functionality:
$html = file_get_contents($url); $doc = new DOMDocument(); $doc->loadHTML($html); $xpath = new DOMXPath($doc); $elements = $xpath->query('//*[@id="resultCount"]/span');
Re: Web Scraping 101 with Python
#70BeautifulSoup, has received a lot of positive press on HN over the years so when I needed to do some heavy scraping I gave it a spin. It was a total disappointment. It's fine if you are scraping a small set of similar pages from a single site but if you are scraping a large number of pages across many sites and esp pages with text encoding other than ascii / utf-8 it choaks so frequently as to be useless. BeautifulSo…
I don't see a use for BeautifulSoup nowadays. I use lxml.etree for everything (with the HTML parser when needed), and do 99% of queries using XPath. It's the best way to do scraping with Python, in my experience.