Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

102TB of New Crawl Data Available

commoncrawl.org

Post reply on HN
Print view View original

31–39 of 39 posts

Re: 102TB of New Crawl Data Available

#31

Post by daivd »

LisaG wrote: post #3
boyter wrote: post #2

I love common crawl, but as I commented before I still want to see a subset available for download, something like the top million sites or something like that. Certainly a few steps of data, say 50GB 100GB and 200GB. I really think a subset like this will increase the value as it would allow people writing search engines (for fun or profit) to suck a copy down locally and work away. Its something I would like to do…

There will be news about a subset sometime next month!

One subset for each TLD would be nice. Or, if you can afford more CPU-power, per language, using a good open language detector.

Re: 102TB of New Crawl Data Available

#32

Post by wodow »

iamtechaddict wrote: post #26

Is there a way we can access the data(small subet say 30-40GB's) without having an AWS account(as it requires a credit card, I'm a student i don't have any) ?

Some of the older data (2009) is available on archive.org: https://archive.org/details/commoncrawl

Re: 102TB of New Crawl Data Available

#33

Post by ldng »

sirsar wrote: post #11

We have switched the metadata files from JSON to WAT files. The JSON format did not allow specifying the multiple offsets to files necessary for the WARC upgrade and WAT files provide more detail. Where can I read more about this?

Section "Resources" of the post you haven't read ?

Re: 102TB of New Crawl Data Available

#34

Post by iamtechaddict »

wodow wrote: post #32
iamtechaddict wrote: post #26

Is there a way we can access the data(small subet say 30-40GB's) without having an AWS account(as it requires a credit card, I'm a student i don't have any) ?

Some of the older data (2009) is available on archive.org: https://archive.org/details/commoncrawl

Thanks a lot. It'll be very helpful i'm sure.

Re: 102TB of New Crawl Data Available

#35

Post by GigabyteCoin »

Can anyone give me a quick rundown on how exactly one gains access to all of this data?

I have heard about this project numerous times, and am always dissuaded by the lack of download links/torrents/information on their homepage.

Perhaps I just don't know what I'm looking at?

Re: 102TB of New Crawl Data Available

#36

Post by wpietri »

GigabyteCoin wrote: post #35

Can anyone give me a quick rundown on how exactly one gains access to all of this data? I have heard about this project numerous times, and am always dissuaded by the lack of download links/torrents/information on their homepage. Perhaps I just don't know what I'm looking at?

Did you try this?

http://commoncrawl.org/get-started/

I haven't tried that one, but I've poked at other of the Amazon Common Datasets collection:

http://aws.amazon.com/datasets

If you're already familiar with using Amazon's virtual servers, it's pretty straightforward.

I also note that the Common Crawl project publishes code here:

https://github.com/commoncrawl/commoncrawl

Re: 102TB of New Crawl Data Available

#38

Post by sirsar »

ldng wrote: post #33
sirsar wrote: post #11

We have switched the metadata files from JSON to WAT files. The JSON format did not allow specifying the multiple offsets to files necessary for the WARC upgrade and WAT files provide more detail. Where can I read more about this?

Section "Resources" of the post you haven't read ?

No, I mean the difference between the filetypes.
Post reply on HN