Viewing profile — ahadrana

Hi, you can view our terms of use at http://www.commoncrawl.org/about/terms-of-use/full-terms-of-... . We adhere to the robots.txt standard, try to do all our crawling above board,…

We hear you. Could you define some criteria as to the type and size of sample data you would like to see? We are working on producing more targeted/limited collections, like perhap…

The pagerank and other metadata we compute is not part of the S3 corpus, but we do collect this information and probably will make it available in a separate S3 bucket in Hadoop Se…

Hi I work at commoncrawl. We have spent our time (in 2011) improving our algorithms, and hopefully this effort will start to show real results (with respect to crawl frequency and …

Sorry, our github repository had some accidental check-ins that we needed to remove. I will share the link to the code shortly.

Hi, I work at commoncrawl, so I will try to answer your question. We store our crawl data on S3 in the form of 100MB compressed archives and there are between 40,000 and 50,000 suc…

Hi. I work for commoncrawl. We are about to start an improved recrawl and will be doing this more frequently going forward. In the process we will also consolidate our data on S3 t…