Ask HN: Has anyone ever crawled over a billion pages? How much did it cost?

Post by mtmail » Tue, Sep 13, 2016, 10:22 PM UTC

Post by mtmail » Tue, Sep 13, 2016, 10:41 PM UTC

outpan wrote: post #3

mtmail wrote: post #2

There's a discussion about a 2 billion page crawl on the frontpage right now. https://news.ycombinator.com/item?id=12486631 Here's the author's comment on hardware https://news.ycombinator.com/item?id=12487003 and later he says it costs 300 Euro/month to run the service.

That post is what triggered my Ask post. The problem is the huge contrast with https://www.quora.com/How-much-would-it-cost-to-crawl-1-bill... Even taking into account the drop in prices on AWS. Also, if you take a quick look at companies that provide such services the prices are orders of magnitude higher than deusu's costs.

Post by detaro » Tue, Sep 13, 2016, 10:50 PM UTC

outpan wrote: post #5

mtmail wrote: post #4

Earlier quoted context omitted.

Deusu's crawl servers are located at https://www.hosteurope.de/en/Server/Root-Server/ while the website points to his home broadband ISP. Two servers at his specs would be 200 Euro/month total, with 5x more bandwidth than he currently uses. I'd say that's much cheaper that AWS. Of course crawl companies charge more: they run a business, pay system administrators, have more backup and redundancy.

I'm not sure how he manages to crawl with this speed using such low amount of resources. We did a benchmark on Nutch and couldn't really pass the 10-14 M(B)ps on a $1200/month machine. Even though we hired a professional to optimize the setup. The same is roughly true about Heritrix. Just wondering if there is something missing in his setup, such as domain/ip rate limiting.

Post by outpan » Wed, Sep 14, 2016, 12:37 AM UTC

AznHisoka wrote: post #7

I've crawled over a billion pages over a stretch of 3 years or so. Crawling is the easy task and just crawling a billion pages wouldn't cost more than a few thousand a month. Add a couple more thousand for storing these pages in a search index and database.

Post by AznHisoka » Wed, Sep 14, 2016, 3:12 PM UTC

outpan wrote: post #8

AznHisoka wrote: post #7

I've crawled over a billion pages over a stretch of 3 years or so. Crawling is the easy task and just crawling a billion pages wouldn't cost more than a few thousand a month. Add a couple more thousand for storing these pages in a search index and database.

Would you be able to share what your stack was? and the resources it took? Thanks a lot.