Can anyone list some good resources about scraping, with gotchas etc.?
I really the scraping chapter in the Bastard's Book of Ruby
http://ruby.bastardsbook.com/chapters/web-scraping/
Web scraping with Ruby
11–20 of 32 posts
Re: Web scraping with Ruby
#12I had to write scrapers in Ruby for a very large application that scraped all kinds of government information from various states. We found (after a lot of pain working with very procedural scrapers) that a modified producer/consumer pattern worked well. We found that making classes for the producers (they were classes that described each page to be scraped, with methods that matched the modeled data) allowed for easy maintenance. We then created consumers that could be passed any of the page specific producer classes, and knew how to persist the scraped data.
Once I had a good pattern in place I could easily create subclasses of the data type I was trying to scrape, basically pointing each of the modeled data methods to an xpath that was specific to that page.
Re: Web scraping with Ruby
#13I'd suggest taking a look at Scrapy (http://scrapy.org). It is built on top of Twisted (asynchronous) and uses xPath which makes your "scraping" code a lot more readable.
Re: Web scraping with Ruby
#14I'd suggest going with mechanize from the off - not just, as the article says, "[when] the site you’re scraping requires you to login first, for those instances I recommend looking into mechanize". Mechanize allows you to write clean, efficient scraper code without all the boilerplate. It's the nicest scraping solution I've yet encountered.
I agree that mechanize is an excellent scraping solution, but for something really basic like this where we're not clicking links or submitting forms it seemed like a bit of an overkill :)
Each to their own, but I find the Mechanize syntax much easier even for simple scraping work. You can use CSS selectors as per the example, or XPath should you want to get more complex.
Re: Web scraping with Ruby
#15I'd suggest taking a look at Scrapy ( http://scrapy.org ). It is built on top of Twisted (asynchronous) and uses xPath which makes your "scraping" code a lot more readable.
Nokogiri can use xpath, as well, FWIW. The article's example could be a lot more terse.
Re: Web scraping with Ruby
#16How do you get the script to save the json file?
Or, you can write it from within Ruby:
open("out.json", "w") {|f| f.puts JSON.dump(showings) }Re: Web scraping with Ruby
#17Why not just use like watir or selenium?
Re: Web scraping with Ruby
#18I recommend having a look at capybara [0]. It is build on top of nokogiri, and is actually a tool to write acceptence tests. But it can also be used for web scraping: you can open websites, click on links, fill in forms, find elements on a page (via xpath or css), get their values, etc... I prefer it over nokogiri because of its nice DSL and good documentation [1]. It also can execute javascript, which sometimes is handy for scraping.
I've spend a lot of time working on web scrapers for two of my projects, http://themescroller.com (dead) and http://www.remoteworknewsletter.com, and I think the holy grail is to build a rails app around your scraper. You can write your scrapers as libs, and then make them executable as rake tasks, or even cronjobs. And because its a rails app you can save all scraped data as actual models and have them persisted in a database. With rails its also super easy to build an api around your data, or build a quick backend for it via rails scaffolds.
[0] https://github.com/jnicklas/capybara [1] http://www.rubydoc.info/github/jnicklas/capybara/
Re: Web scraping with Ruby
#19Re: Web scraping with Ruby
#20I had to write scrapers in Ruby for a very large application that scraped all kinds of government information from various states. We found (after a lot of pain working with very procedural scrapers) that a modified producer/consumer pattern worked well. We found that making classes for the producers (they were classes that described each page to be scraped, with methods that matched the modeled data) allowed for eas…
We do something very similar & I'd love to get in touch if you'd be interested in discussing further. My email is in my profile if you'd be willing to reach out.