Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Free 5 Billion Page Web Index Now Available from Common Crawl Foundation

readwriteweb.com

Post reply on HN
Print view View original

31–40 of 42 posts

Re: Free 5 Billion Page Web Index Now Available from Common Crawl Foundation

#31

Post by showerst »

ahadrana wrote: post #30
wisty wrote: post #29

Is there a sample dataset? I think all projects should have sample datasets. It simplifies a lot of things, and in this case stops hundreds of geeks burning through bandwidth before they realize they don't have a clue what they are going to do with the data.

We hear you. Could you define some criteria as to the type and size of sample data you would like to see? We are working on producing more targeted/limited collections, like perhaps all most recently published blog posts etc.

Perhaps two sets, one that's just a few hundred kilobytes that contains a few sample .arc files to test against the format, and then one larger 'training' set that's small enough to test against offline (maybe like 100MB?) but large enough to contain a good sample of the possible content.

Re: Free 5 Billion Page Web Index Now Available from Common Crawl Foundation

#32

Post by ohashi »

dotBen wrote: post #26

Although I'm personally all for open distribution of crawl data like this and all of my personal websites are CC-licensed, isn't there something to be said for the copyright status of the pages in the crawl file? The crawl file presumably contains the contents of websites and so the owners of those websites could assert that Common Crawl Foundation is distributing their work without permission or license. There are a…

I see it in the ToS:

http://www.commoncrawl.org/about/terms-of-use/

-Violate other people’s rights (IP, proprietary, etc.)

Re: Free 5 Billion Page Web Index Now Available from Common Crawl Foundation

#33

Post by rgrieselhuber »

> We do not use Nutch for the purposes of crawling, but instead utilize a custom crawl infrastructure to strictly limit the rate at which we crawl individual web hosts.

Were there any other reasons to not use Nutch (performance, etc.)?

I'd love to hear more about the stack you're using to perform the crawls. If you don't mind sharing, it would be very interesting to read about the costs involved in gathering this data (how many machines, how long did it take, etc.)

Any plans to open source that as well? In addition to a general lack of open web crawl data freely available, there are precious few open source projects (if any) that produce high quality crawlers able to deal with the modern web.

Re: Free 5 Billion Page Web Index Now Available from Common Crawl Foundation

#34

Post by rb2k_ »

Oh nice. I've been doing a lot of crawling myself (http://blog.marc-seeger.de/2010/12/09/my-thesis-building-blo...) and I'd love to get my hands on this data. I hope they'll segment their data a bit further.

I personally would LOVE to have a simple list of the domainnames themselves without all of the connections and documents.

Also: Why not just use bittorrent to distribute it?

Re: Free 5 Billion Page Web Index Now Available from Common Crawl Foundation

#35

Post by michels24 »

Aloisius wrote: post #3
pooyak wrote: post #2

One interesting discussion from here: http://www.commoncrawl.org/common-crawl-enters-a-new-phase/ It says the cost of running a hadoop job to scan all 5billon documents is in the order of $100. Does any one know how does this compare to let say Yahoo BOSS? Is it even comparable?

Does BOSS still exist? I was under the impression that it was defunct.

I was the former GM of Yahoo BOSS (was there from pre-launch through 11/09). BOSS does still exist - http://developer.yahoo.com/search/boss/. It is now a paid API under the umbrella of Yahoo Developer Network. The pricing plan (http://developer.yahoo.com/search/boss/#pricing) is based on query type and volume. Unfortunately there is no self-serve advertising model (meaning if you incorporate Y!/Bing search ads, the service is free). It's important to note though that this is the Bing search index, not the old Yahoo Search index that is effectively shut down. The original BOSS product was based on Yahoo! Search.

From what I have heard BOSS continues to do very well and is pointed at internally as how to turn an API into a real business and product.

One more note, I am now at Factual where we are very happy consumers of the CommonCrawl service.

Re: Free 5 Billion Page Web Index Now Available from Common Crawl Foundation

#36

Post by pablohoffman »

I initially submitted this post, but then deleted it and resubmitted to the original post on Common Crawl blog: http://news.ycombinator.com/item?id=3208853

I now regret since this one got much more attention. I was under the impression that linking to the original post was more welcomed here HN, but it seems this is not always the case.

Re: Free 5 Billion Page Web Index Now Available from Common Crawl Foundation

#37

Post by Aloisius »

rb2k_ wrote: post #34

Oh nice. I've been doing a lot of crawling myself ( http://blog.marc-seeger.de/2010/12/09/my-thesis-building-blo... ) and I'd love to get my hands on this data. I hope they'll segment their data a bit further. I personally would LOVE to have a simple list of the domainnames themselves without all of the connections and documents. Also: Why not just use bittorrent to distribute it?

I imagine they don't use bittorrent because it is both very large (TBs) and changes frequently.

With S3, you could boot up a bunch of Hadoop processes, pull it (without incurring any bandwidth costs I believe), process it and dump out whatever you want.

Re: Free 5 Billion Page Web Index Now Available from Common Crawl Foundation

#39

Post by ahadrana »

dotBen wrote: post #26

Although I'm personally all for open distribution of crawl data like this and all of my personal websites are CC-licensed, isn't there something to be said for the copyright status of the pages in the crawl file? The crawl file presumably contains the contents of websites and so the owners of those websites could assert that Common Crawl Foundation is distributing their work without permission or license. There are a…

Hi, you can view our terms of use at http://www.commoncrawl.org/about/terms-of-use/full-terms-of-.... We adhere to the robots.txt standard, try to do all our crawling above board, and (strictly personal opinion here) we are definitely not in the business of diminishing or subverting peoples rights with regards to the content they produce. There are many other options available to those who are determined to crawl a site's content, whether the site owner wants them to or not. Our goal is to democratize access to our crawl for the betterment of Web ecosystem as a whole and we believe storing the data on S3 and making it accessible to a wide audience is the right way to accomplish this goal.

Re: Free 5 Billion Page Web Index Now Available from Common Crawl Foundation

#40

Post by dcnstrct »

showerst wrote: post #31
ahadrana wrote: post #30

Earlier quoted context omitted.

We hear you. Could you define some criteria as to the type and size of sample data you would like to see? We are working on producing more targeted/limited collections, like perhaps all most recently published blog posts etc.

Perhaps two sets, one that's just a few hundred kilobytes that contains a few sample .arc files to test against the format, and then one larger 'training' set that's small enough to test against offline (maybe like 100MB?) but large enough to contain a good sample of the possible content.

Concur with this comment -- it might also help the community provide feedback on structure and ways to segment that data so that there are more directed efforts to consume small parts of the crawl for processing
Post reply on HN