11–18 of 18 posts
You'll likely find this tool handy once you choose your framework: http://www.selectorgadget.com/
Another quick and dirty way: http://googlemapsmania.blogspot.com/2008/10/data-scaping-wik...
Earlier quoted context omitted.
I've never written anything like this... Is there a function like CURL that downloads the page as a string from a URL?
require 'net/http' require 'uri' output = Net::HTTP.get(URI.parse("http://www.google.com"))
require 'curb' c = Curl::Easy.perform("http://www.google.com) puts c.body_str
Anemone is great because it can focus your crawl to only url's that match a certain pattern, which really helps you traverse a small portion of a larger website (like a University site). You can also do specific actions on pages that match a certain pattern.
For scraping, anemone natively supports nokogiri, so since you're coming from a blank slate, it might be easiest to learn nokogiri. Before discovering anemone, I had already written what needed to be done on each page in hpricot, so my code is a bit messy, but it's not that difficult to get anemone and hpricot to work together.
So please, use nokogiri it's a great library and the only thing I really miss from ruby-land in python.
Earlier quoted context omitted.
require 'net/http' require 'uri' output = Net::HTTP.get(URI.parse("http://www.google.com"))
curb is a lot faster than net/http require 'curb' c = Curl::Easy.perform(" http://www.google.com ) puts c.body_str
Earlier quoted context omitted.
curb is a lot faster than net/http require 'curb' c = Curl::Easy.perform(" http://www.google.com ) puts c.body_str
So apparently accessing a secure URL isn't straightforward... No login is required, its just an HTTPS but it won't go in curb.
Earlier quoted context omitted.
require 'net/http' require 'uri' output = Net::HTTP.get(URI.parse("http://www.google.com"))
curb is a lot faster than net/http require 'curb' c = Curl::Easy.perform(" http://www.google.com ) puts c.body_str