A Look Inside Our 210TB 2012 Web Corpus

#1

Post by LisaG » Tue, Aug 13, 2013, 10:10 PM UTC

A Look Inside Our 210TB 2012 Web Corpus
commoncrawl.org

Re: A Look Inside Our 210TB 2012 Web Corpus

#2

Post by Aloisius » Tue, Aug 13, 2013, 10:33 PM UTC

Link to the PDF mentioned: https://docs.google.com/file/d/1_9698uglerxB9nAglvaHkEgU-iZN...

Re: A Look Inside Our 210TB 2012 Web Corpus

#3

Post by spimmy » Tue, Aug 13, 2013, 10:49 PM UTC

What do you mean by "open"? Can the data be used for startups and other commercial purposes?

Re: A Look Inside Our 210TB 2012 Web Corpus

#4

Post by res0nat0r » Tue, Aug 13, 2013, 10:53 PM UTC

spimmy wrote: post #3

What do you mean by "open"? Can the data be used for startups and other commercial purposes?

The data is freely available: http://aws.amazon.com/datasets/41740

and you just need to comply with the Common Crawl TOU: http://commoncrawl.org/about/terms-of-use/

Re: A Look Inside Our 210TB 2012 Web Corpus

#5

Post by CrazedGeek » Tue, Aug 13, 2013, 10:53 PM UTC

spimmy wrote: post #3

What do you mean by "open"? Can the data be used for startups and other commercial purposes?

From the FAQ: "Please refer to the Common Crawl Terms of Use document for a detailed, authoritative description of our Terms of Use guidelines, but, in general, you cannot republish the data retrieved from the crawl (unless allowed by fair use), you cannot resell access to the service, you cannot use the crawl data for any illegal purposes, and you must respect the Terms of Use of the sites we crawl."

http://commoncrawl.org/about/terms-of-use/

Re: A Look Inside Our 210TB 2012 Web Corpus

#6

Post by natch » Tue, Aug 13, 2013, 10:54 PM UTC

How does one get set up to access the s3:// links their blog posts reference? I do realize these point to Amazon S3 buckets, but how to get at them?

Re: A Look Inside Our 210TB 2012 Web Corpus

#7

Post by Aloisius » Tue, Aug 13, 2013, 10:55 PM UTC

spimmy wrote: post #3

What do you mean by "open"? Can the data be used for startups and other commercial purposes?

Yes! Startups/commercial companies/etc can all use the data for free. The terms of use basically say, don't do anything illegal with it and a few other things, but it shouldn't affect the vast majority of uses.

Actually, tomorrow a video on a startup that uses Common Crawl data is getting posted.

Re: A Look Inside Our 210TB 2012 Web Corpus

#8

Post by rgrieselhuber » Tue, Aug 13, 2013, 10:55 PM UTC

Is there something, other than funding, preventing a more regular, open-sourced crawl of the web?

Re: A Look Inside Our 210TB 2012 Web Corpus

#9

Post by Aloisius » Tue, Aug 13, 2013, 10:58 PM UTC

natch wrote: post #6

How does one get set up to access the s3:// links their blog posts reference? I do realize these point to Amazon S3 buckets, but how to get at them?

You need an Amazon account - though the data is available for free, I think you need to specify your access key to actually fetch it.

From there you can grab the S3 command line tools (http://s3tools.org/s3cmd) or load it up from hadoop or through one of the various open source libraries (boto for instance).

Re: A Look Inside Our 210TB 2012 Web Corpus

#10

Post by LisaG » Tue, Aug 13, 2013, 11:02 PM UTC

rgrieselhuber wrote: post #8

Is there something, other than funding, preventing a more regular, open-sourced crawl of the web?

Limited resources are the only reason. We are working on a subset crawl of ~3 million pages that will be published weekly starting two weeks from now. But doing the full crawl takes a lot of time, effort and money.