Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

102TB of New Crawl Data Available

commoncrawl.org

Post reply on HN
Print view View original

21–30 of 39 posts

Re: 102TB of New Crawl Data Available

#21

Post by ccleve »

Aloisius wrote: post #13
hkmurakami wrote: post #7

Earlier quoted context omitted.

would love to have even smaller subsets (like 5gb) that students can casually play around with too to practice and learn tools and algos :) (if it's not too much trouble!)

You can fetch a single WARC file directly like say: s3://aws-publicdatasets/common-crawl/crawl-data/CC-MAIN-2013-20/segments/1368704392896/warc/CC-MAIN-20130516113952-00058-ip-10-60-113-184.ec2.internal.warc.gz They are around 850 MB each. The text extracts and metadata files are generated off individual WARC files, so it is pretty easy to get the corresponding sets of files. For the above it would be: s3://aws-publi…

Is there any way to get incrementals? It would be extremely valuable is to get the pages that were added/changed/deleted each day. Some kind of a daily feed of a more limited size.

Re: 102TB of New Crawl Data Available

#22

Post by nl »

malandrew wrote: post #16
LisaG wrote: post #3

Earlier quoted context omitted.

There will be news about a subset sometime next month!

Ideally beyond the top sites, these subsets would be available as verticals, so that people can focus on specialized search engines. While it's nice to have generalist search engines, it would be even better to be able to unbundle the generalist search engines completely. Verticals such as the following would be nice: 1) Everything linux, unix and both 2) Everything open-source 3) Only news & current events 4) Popula…

You don't usually download this data - you process it on AWS to your requirements.

Seriously - they give you an easy way to create these subsets yourself[1]. That is a much better solution than them trying to anticipate the exact needs of every potential client.

[1] http://commoncrawl.org/get-started/

Re: 102TB of New Crawl Data Available

#23

Post by froo »

ccleve wrote: post #21
Aloisius wrote: post #13

Earlier quoted context omitted.

You can fetch a single WARC file directly like say: s3://aws-publicdatasets/common-crawl/crawl-data/CC-MAIN-2013-20/segments/1368704392896/warc/CC-MAIN-20130516113952-00058-ip-10-60-113-184.ec2.internal.warc.gz They are around 850 MB each. The text extracts and metadata files are generated off individual WARC files, so it is pretty easy to get the corresponding sets of files. For the above it would be: s3://aws-publi…

Is there any way to get incrementals? It would be extremely valuable is to get the pages that were added/changed/deleted each day. Some kind of a daily feed of a more limited size.

  s3cmd ls s3://aws-publicdatasets/common-crawl/crawl-data/CC-MAIN-2013-20/segments/
That should get you about 90% on your way.

Re: 102TB of New Crawl Data Available

#25

Post by malandrew »

nl wrote: post #22
malandrew wrote: post #16

Earlier quoted context omitted.

Ideally beyond the top sites, these subsets would be available as verticals, so that people can focus on specialized search engines. While it's nice to have generalist search engines, it would be even better to be able to unbundle the generalist search engines completely. Verticals such as the following would be nice: 1) Everything linux, unix and both 2) Everything open-source 3) Only news & current events 4) Popula…

You don't usually download this data - you process it on AWS to your requirements. Seriously - they give you an easy way to create these subsets yourself[1]. That is a much better solution than them trying to anticipate the exact needs of every potential client. [1] http://commoncrawl.org/get-started/

I guess what I was suggesting is "given enough eyeballs, all spam and poor quality content is shallow"

There is definitely a benefit in using the community to identify valuable subsets and then individually putting your energy towards building discovery/search products around that subset.

Re: 102TB of New Crawl Data Available

#28

Post by gsnedders »

nl wrote: post #22
malandrew wrote: post #16

Earlier quoted context omitted.

Ideally beyond the top sites, these subsets would be available as verticals, so that people can focus on specialized search engines. While it's nice to have generalist search engines, it would be even better to be able to unbundle the generalist search engines completely. Verticals such as the following would be nice: 1) Everything linux, unix and both 2) Everything open-source 3) Only news & current events 4) Popula…

You don't usually download this data - you process it on AWS to your requirements. Seriously - they give you an easy way to create these subsets yourself[1]. That is a much better solution than them trying to anticipate the exact needs of every potential client. [1] http://commoncrawl.org/get-started/

Is the example code still right with the new file formats for this new crawl?
Post reply on HN