Web scraping with Ruby

#1

Post by hecticjeff » Mon, Jan 19, 2015, 9:08 AM UTC

Re: Web scraping with Ruby

#2

Post by Doctor_Fegg » Mon, Jan 19, 2015, 11:39 AM UTC

I'd suggest going with mechanize from the off - not just, as the article says, "[when] the site you’re scraping requires you to login first, for those instances I recommend looking into mechanize".

Mechanize allows you to write clean, efficient scraper code without all the boilerplate. It's the nicest scraping solution I've yet encountered.

Re: Web scraping with Ruby

#3

Post by mychaelangelo » Mon, Jan 19, 2015, 11:42 AM UTC

thanks for sharing this - great scraping intro for us newbies (I'm new to ruby and ROR).

Re: Web scraping with Ruby

#4

Post by richardpetersen » Mon, Jan 19, 2015, 11:54 AM UTC

How do you get the script to save the json file?

Re: Web scraping with Ruby

#5

Post by hecticjeff » Mon, Jan 19, 2015, 11:58 AM UTC

richardpetersen wrote: post #4

How do you get the script to save the json file?

You can redirect the output of the script to a json file, so in this case something like:

$ ruby scraper.rb > showings.json

Re: Web scraping with Ruby

#6

Post by hecticjeff » Mon, Jan 19, 2015, 11:59 AM UTC

Doctor_Fegg wrote: post #2

I'd suggest going with mechanize from the off - not just, as the article says, "[when] the site you’re scraping requires you to login first, for those instances I recommend looking into mechanize". Mechanize allows you to write clean, efficient scraper code without all the boilerplate. It's the nicest scraping solution I've yet encountered.

I agree that mechanize is an excellent scraping solution, but for something really basic like this where we're not clicking links or submitting forms it seemed like a bit of an overkill :)

Re: Web scraping with Ruby

#7

Post by k__ » Mon, Jan 19, 2015, 12:10 PM UTC

Can anyone list some good resources about scraping, with gotchas etc.?

Re: Web scraping with Ruby

#8

Post by richardpetersen » Mon, Jan 19, 2015, 12:11 PM UTC

hecticjeff wrote: post #5

richardpetersen wrote: post #4

How do you get the script to save the json file?

You can redirect the output of the script to a json file, so in this case something like: $ ruby scraper.rb > showings.json

Perfect, thank you

Re: Web scraping with Ruby

#9

Post by forlorn » Mon, Jan 19, 2015, 1:11 PM UTC

k__ wrote: post #7

Can anyone list some good resources about scraping, with gotchas etc.?

My recipe is to use Typhoeus (https://github.com/typhoeus/typhoeus) + Nokogiri. I have tried lots of different options including EventMachine with em-http-request and reactor loop and concurrent-ruby (both a re very poorly documented)

Typhoeus has a built-in concurrency mechanism with callbacks with specified number of concurrent http requests. You just create a hydra object, create the first request object with URL and a callback (you have to check errors like 404 yourself) where you extract another URLs from the page and push them to hydra again with the same on another callback.

Re: Web scraping with Ruby

#10

Post by Jake232 » Mon, Jan 19, 2015, 1:54 PM UTC

k__ wrote: post #7

Can anyone list some good resources about scraping, with gotchas etc.?

I wrote an article on scraping last year which recieved a lot of praise. May be worth a read - http://jakeaustwick.me/python-web-scraping-resource/