The Web is missing an essential part of infrastructure: an open web index

Abstract:

A proposal for building an index of the Web that separates the infrastructure part of the search engine - the index - from the services part that will form the basis for myriad search engines and other services utilizing Web data on top of a public infrastructure open to everyone.

whole document here from arxiv.org page :

https://arxiv.org/pdf/1903.03846 [PDF]

Isn't this what Common Crawl[1] is. From their FAQ:

> What is Common Crawl?

> Common Crawl is a 501(c)(3) non-profit organization dedicated to providing a copy of the internet to internet researchers, companies and individuals at no cost for the purpose of research and analysis.

> What can you do with a copy of the web?

> The possibilities are endless, but people have used the data to improve language translation software, predict trends, track the disease propagation and much more.

> Can’t Google or Microsoft just do that?

>Our goal is to democratize the data so everyone, not just big companies, can do high quality research and analysis.

Also DuckDuckGo founder Gabriel Weinberg expressed the sentiment that the index should be separate from the search engine many years ago:

> Our approach was to treat the “copy the Internet” part as the commodity. You could get it from multiple places. When I started, Google, Yahoo, Yandex and Microsoft were all building indexes. We focused on doing things the other guys couldn’t do. [2]

From what I remember reading once DuckDuckGo doesn't use Common Crawl though.

[1] https://commoncrawl.org/

[2] https://www.japantimes.co.jp/news/2013/07/28/business/duckdu...

I dont believe Common Crawl offers a real time search index as its delayed by more than a month (although that could have changed recently). Still useful for research purposes but not that desirable for a search engine that competes with Google, etc.

It won't work without a central authority. See Soundcloud as an example. People tag their music with whatever they think will get them traffic. So, in order to do this you'll need a mass of volunteers which will lead to politics "XYZ should be classified as G! No it should be F!", "classifying ABC as DEF is racists/sexist/..." and other arguments. You'll also get people lobbying to have things removed (right to be forgotten, pornography, drug ad, prostitution ads, disparaging the government -China, -Thailand, etc..., etc...

I'm not saying it shouldn't be done but I think it will be way more work than expected and there will be all kinds of issues.