Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Look Inside Our 210TB 2012 Web Corpus

commoncrawl.org

Post reply on HN
Print view View original

31–38 of 38 posts

Re: A Look Inside Our 210TB 2012 Web Corpus

#31

Post by greglindahl »

gojomo wrote: post #27
wicknicks wrote: post #20

Earlier quoted context omitted.

Good crawlers should typically avoid wikipedia links, to avoid the number of HTTP requests on wiki servers (and keep their costs down), esp. because they make available whole data dumps for download through a separate cheaper channel: http://en.wikipedia.org/wiki/Wikipedia:Database_download

Yes and no. Some crawlers are most interested in freshest versions of the most inlinked articles, or in the exact HTML presentation at Wikipedia. The monthly full raw wikitext dumps don't provide that. And, Wikipedia's serving plant is pretty efficient, with bandwidth only being a small portion of their costs. They can afford some crawling... and correspondingly, their /robots.txt is pretty open. Good crawlers seekin…

blekko attempted to implement crawling a local copy, and it was a PITA. We'd rather crawl the real thing with a crawl-delay of 1. Best would be if the Wikimedia Foundation made a .html dump available.

Re: A Look Inside Our 210TB 2012 Web Corpus

#32

Post by jamesaguilar »

sylvinus wrote: post #13

Common Crawl is awesome. I wonder how complex it would be to run a Google-like frontend on top of it, and how good the results would be after a couple days of hacking...

Not very, or else someone would have replicated Google already (with fewer engineers and less money than Microsoft has thrown at the problem).

Re: A Look Inside Our 210TB 2012 Web Corpus

#33

Post by sytelus »

sylvinus wrote: post #13

Common Crawl is awesome. I wonder how complex it would be to run a Google-like frontend on top of it, and how good the results would be after a couple days of hacking...

This is actually very small corpus, just 3 billion docs. Google, for example, is known to have 50 billion docs in index.

Re: A Look Inside Our 210TB 2012 Web Corpus

#34

Post by asgard1024 »

boyter wrote: post #18
sylvinus wrote: post #13

Common Crawl is awesome. I wonder how complex it would be to run a Google-like frontend on top of it, and how good the results would be after a couple days of hacking...

Very and probably not very good (Compare Gigablast to Google as an example of why its hard). Not to take anything away from Common Crawl but crawling is often one of the easier things to build when creating a search engine. A crawler can be as simple as for(listofurls) { geturl; add urls to listofurls; } Doing it on a large scale over and over is a harder problem (which common crawl does for you) but its not too diff…

Actually, not so simple. Sure, you can do simple crawling easily; but the hard part is to extract meaningful data from it. It's very easy to loop on many sites for instance. Protocol violations abound - some sites serve binaries as text/html, for instance.

What I heard about a smaller search engine was that web crawling is usually augmented with some manually added rules for various sites to prevent spoiling the database. Not a trivial task at all.

Doing queries is IMHO algorithmically much better understood, because it's a constrained problem. But getting information extracted out from the real world, with all the PHP and HTML "hackers", not so easy.

Re: A Look Inside Our 210TB 2012 Web Corpus

#35

Post by Sven7 »

asgard1024 wrote: post #34
boyter wrote: post #18

Earlier quoted context omitted.

Very and probably not very good (Compare Gigablast to Google as an example of why its hard). Not to take anything away from Common Crawl but crawling is often one of the easier things to build when creating a search engine. A crawler can be as simple as for(listofurls) { geturl; add urls to listofurls; } Doing it on a large scale over and over is a harder problem (which common crawl does for you) but its not too diff…

Actually, not so simple. Sure, you can do simple crawling easily; but the hard part is to extract meaningful data from it. It's very easy to loop on many sites for instance. Protocol violations abound - some sites serve binaries as text/html, for instance. What I heard about a smaller search engine was that web crawling is usually augmented with some manually added rules for various sites to prevent spoiling the data…

Which is one of the main reasons Google has no serious competition in search except possibly in China.

It is also why the rate of innovation in search isn't moving as fast as it can be moving.

If Google opened up (unlimited) web API access to their search interface, to say a large city for a year or two people would really get a taste of what innovation in search looked like.

And of course it would be in Google's interest cause search as a platform or marketplace is where the future of Google really lies. All the other advertising empire defending distractions like Android, Chrome and YouTube are really sideshows.

Re: A Look Inside Our 210TB 2012 Web Corpus

#36

Post by boyter »

asgard1024 wrote: post #34
boyter wrote: post #18

Earlier quoted context omitted.

Very and probably not very good (Compare Gigablast to Google as an example of why its hard). Not to take anything away from Common Crawl but crawling is often one of the easier things to build when creating a search engine. A crawler can be as simple as for(listofurls) { geturl; add urls to listofurls; } Doing it on a large scale over and over is a harder problem (which common crawl does for you) but its not too diff…

Actually, not so simple. Sure, you can do simple crawling easily; but the hard part is to extract meaningful data from it. It's very easy to loop on many sites for instance. Protocol violations abound - some sites serve binaries as text/html, for instance. What I heard about a smaller search engine was that web crawling is usually augmented with some manually added rules for various sites to prevent spoiling the data…

Personally I consider extracting meaning from the crawl part of the indexing step. That just comes down to how you define it though. In reality its blurry as you need to do some pre-indexing during the crawl to extract meaningful data and as you say there are a lot of edge cases.

For basic crawling it really is as simple as while links download link though.

Re: A Look Inside Our 210TB 2012 Web Corpus

#37

Post by mark_l_watson »

wicknicks wrote: post #21
mark_l_watson wrote: post #12

Check out the Common Crawl contest winning projects from the linked page - some very good work, and a good source of ideas and techniques: http://commoncrawl.org/the-winners-of-the-norvig-web-data-sc... Some good stuff!

I loved the inter-lingual web page linkage visualization project. Any idea why Traitor won the contest? It seems very similar to regular "create inverted index with map reduce" problem, or am I missing something?

Perhaps Traitor won because it is such a good example of using Map Reduce over the Common Crawl data? I agree that inter-lingual was a cool project.

Re: A Look Inside Our 210TB 2012 Web Corpus

#38

Post by dsinha »

LisaG wrote: post #24
boyter wrote: post #22

Earlier quoted context omitted.

Is that really worth it though? I can crawl 3 million pages in less than 24 hours without any real effort on my part. Or are you going to provide 3 million of the most useful pages? Depth or breadth first crawl?

We do think it is worth it to avoid duplicative efforts. Suppose you crawl 3 million pages and you pay for the compute and storage costs. Then the next person who wants crawl data goes through the same effort and pays the same costs. Doesn't it make much more sense to have a common pool of open data that everyone can use? Even if the effort and costs are low, they are not zero. For the smaller frequent crawl, we are…

Isn't there also the additional factor that webservers sometimes allow only the major search engines to crawl? If so, with something like this, should it gain popularity, and as more apps start using it, you'd hope more webservers allow the common crawler to crawl their websites which they might not if everyone were doing it individually...thinking aloud...
Post reply on HN