How would you build an internet scale web crawler? • Hacker News Board

Re: How would you build an internet scale web crawler?

#11

Post by patrickxie » Fri, Apr 06, 2018, 7:49 PM UTC

tgamba wrote: post #8

I worked at Alexa, then affiliatated with the Internet Archive, on exactly this, in the late 90s. We built our own server farm to crawl, process and store the data. We had 30TB of storage, holding three "snapshots" of the web, and thought we were pretty hot stuff. That would sit on your desktop today. Crawling was the easy part. We had two processes of up to 40 threads each bringing the data down. Even this we had to…

impressive thanks for the wonderful read.

Re: How would you build an internet scale web crawler?

#12

Post by xstartup » Sat, Apr 07, 2018, 7:08 AM UTC

One of the major problems is removing duplicate or near duplicate content like images, text etc....

Re: How would you build an internet scale web crawler?

#13

Post by usgroup » Sat, Apr 07, 2018, 2:26 PM UTC

Have a look at common-crawl. They are well documented.

Re: How would you build an internet scale web crawler?

#14

Post by ismail » Sat, Apr 07, 2018, 11:39 PM UTC

tgamba wrote: post #8

I worked at Alexa, then affiliatated with the Internet Archive, on exactly this, in the late 90s. We built our own server farm to crawl, process and store the data. We had 30TB of storage, holding three "snapshots" of the web, and thought we were pretty hot stuff. That would sit on your desktop today. Crawling was the easy part. We had two processes of up to 40 threads each bringing the data down. Even this we had to…

“Alexa could have been Google, but went in another direction.“

Curious, why did they go in another direction?

Re: How would you build an internet scale web crawler?

#15

Post by ThePhysicist » Sun, Apr 08, 2018, 7:32 PM UTC

Concerning the crawling aspect of this, Michael Nielsen wrote a great article about how to download/crawl a significant part of the Internet in 40 hours:

http://www.michaelnielsen.org/ddi/how-to-crawl-a-quarter-bil...

It's a bit dated (from 2012) but probably still relevant.

Re: How would you build an internet scale web crawler?

#16

Post by tgamba » Mon, Apr 09, 2018, 4:29 PM UTC

ismail wrote: post #14

tgamba wrote: post #8

I worked at Alexa, then affiliatated with the Internet Archive, on exactly this, in the late 90s. We built our own server farm to crawl, process and store the data. We had 30TB of storage, holding three "snapshots" of the web, and thought we were pretty hot stuff. That would sit on your desktop today. Crawling was the easy part. We had two processes of up to 40 threads each bringing the data down. Even this we had to…

“Alexa could have been Google, but went in another direction.“ Curious, why did they go in another direction?

Amazon's stated reason for acquiring Alexa was to use Alexa's technology to build a recommendation engine. Search was never a priority for Alexa itself. We are acquired for $100 million, so it was take the money and run.