Webcrawling at Scale with Nokogiri and IronWorker: Part 2

#1

Post by kenfromm » Mon, Sep 17, 2012, 5:12 PM UTC

Webcrawling at Scale with Nokogiri and IronWorker: Part 2
blog.iron.io

Re: Webcrawling at Scale with Nokogiri and IronWorker: Part 2

#2

Post by netvarun » Mon, Sep 17, 2012, 5:59 PM UTC

Might be of interest: I wrote a post a couple of weeks back regarding our distributed crawling architecture built using perl+redis+gearman

How We Built Our 60-Node (Almost) Distributed Web Crawler http://hackerne.ws/item?id=4469911

Re: Webcrawling at Scale with Nokogiri and IronWorker: Part 2

#3

Post by carimura » Mon, Sep 17, 2012, 6:45 PM UTC

netvarun wrote: post #2

Might be of interest: I wrote a post a couple of weeks back regarding our distributed crawling architecture built using perl+redis+gearman How We Built Our 60-Node (Almost) Distributed Web Crawler http://hackerne.ws/item?id=4469911

Awesome. Love the options.

FWIW - the value prop of IronWorker is never having to deal with servers again -- and only pay for the seconds you're actually crawling. Fire up a million crawlers (workers) and it's auto distributed across large sets of machines behind the scenes (no "spin up tear down" either).

So in essence, it's a completely hosted version of what you described. The power is seen by "just trying it". No software (except for your code), no installations, no servers, etc.