What did you discover.
How did you crawl so many sites, how did you discover them, search engine, ip ranges or another method?
I just finished crawling 5.19B web pages, Ask Me Anything
11–20 of 20 posts
Re: I just finished crawling 5.19B web pages, Ask Me Anything
#12How much did it cost?
Re: I just finished crawling 5.19B web pages, Ask Me Anything
#13what did you do to avoid winding up in endless GET url loops? How deep did you get per site, and how did you schedule followup requests?
Re: I just finished crawling 5.19B web pages, Ask Me Anything
#14What tools did you use? What had to be custom-written and why?
Re: I just finished crawling 5.19B web pages, Ask Me Anything
#15What tools did you use? What had to be custom-written and why?
We tried a bunch of technologies like Nutch, Heritrix, Storm Crawler, ... eventually settled on Mixnode and since it's a 'cloud platform' we didn't really have to change anything.
As for processing the data we crawled, we are using ArchiveSpark (https://github.com/helgeho/ArchiveSpark)
Also, Mixnode defaults on Amazon S3 for storage which was ok with us since we're using EC2 for processing the results.
Re: I just finished crawling 5.19B web pages, Ask Me Anything
#16Why didn't you use common crawl instead?
For our purposes Common Crawl's corpus was missing too many websites (possibly due to robots.txt configs of websites) Also we needed some deep coverage which CC could not provide.
Re: I just finished crawling 5.19B web pages, Ask Me Anything
#17what did you do to avoid winding up in endless GET url loops? How deep did you get per site, and how did you schedule followup requests?
Loop/spam prevention was done by mixnode, I'm not sure how they do it.
The data does not follow a DFS or BFS pattern so pages/site varies greatly by a host's server capacity and anti-crawling configs.
There was a minimum of 10 seconds between followup requests to the same website unless robots.txt had a lower delay. Pretty polite...
Re: I just finished crawling 5.19B web pages, Ask Me Anything
#18If you're rate-limited, you can contact the mods via the Contact link in the footer.
Re: I just finished crawling 5.19B web pages, Ask Me Anything
#19How much did it cost?
It was an all inclusive deal:
420 TB at 0.06 per GB = $25,804