The Web is missing an essential part of infrastructure: an open web index

Books, games, hardware etc are all sold for several years.

What you’re describing is just news on the latest updates representing a small slice of the market. Making the remander far from useless.

The code for doing the indexing (at least by default) could be built right in to the web server, so it'd just be a matter of enabling an option in Apache or the like.

It would be pretty easy to verify whether or not the index is accurate with a small random sample of pages on the site, and then penalize / exclude (or do a de-novo crawl) for those sites not providing a legit index.

Maybe https://www.yacy.net?

The web needs to be forked into two distinct standards: One for dynamic content, and one for documents. The first would use basically everything in the HTML5/CSS/JS toolbox, and the second would be more akin to AMP, but for all docs.

The benefits of this would be a standard for Wysiwyg editors (goodbye million rich text editor projects, Markdown and even Microsoft Word), and more semantic markup for both search engines and accessibility.

Right now it takes millions of man hours to create a performant browser, which is limiting those engines to only the largest organizations. Even Microsoft gave up making their own. And even with all that effort, I still can't create a clean HTML document with an interface as rich as MS Word, or even add bold or color formatting to a Twitter post, or update a Wikipedia page without knowing wiki markup.

We need to pull the dynamic, JS powered side of the web out from the core, limit CSS to non-dynamic properties, and standardize on an efficient in-document binary storage akin to MIME email attachments so HTML docs can be self-contained like a Word or PDF doc.

This document-centric web could be marked off within a standard web page, so you could combine it in regular interfaces for things like social network posts. Or it could be self standing, allowing relatively large sites to be created with indexes, footnotes, etc., but served from a basic static server.

This isn't a technical challenge, it's an organizational one. I've thought for years that Mozilla should be doing this, instead of messing with IoT and phones, etc. It's such an obvious problem that needs addressing, and would have a huge payback in terms of advancing the web as we know it.

I think there are a few factors that would make this idea unworkable. There are two categories of issues, technical and economic, that prevent this from working. I'll go into more detail about the technical issues.

The top-level problem is fan-out. If you want to fan the query to the top million domains (far too few to match Google's retrieval depth, but enough to demonstrate the issue), you're going to need to implement some sort of multi-level fanout, since just one machine can't send enough HTTP requests -- nor even establish that many connections -- in a reasonable time-frame. There are going to be severe tail latency issues that will prevent you from gathering documents from potentially relevant sites. You will have to make frustrating tradeoffs about when to time-out per domain queries to provide a good user experience. And many more issues besides. All decisions that are unnecessary if you control the index. Also, internet bandwidth isn't cheap, and you're going to need a lot of it just to consume the top ten results from a million sites.

The next technical issue is that the inverted index is only a small part of what goes into information retrieval. Scoring is at least as important. Modern scorers are multi-level, meaning they do one pass over many documents on a simple, correlated representation of the data. Then they do a second pass on a more comprehensive form (i.e. the whole document, and metadata about the document), but over fewer documents. There can even be third and fourth passes. The data for the first pass is often embedded directly into the index, and it would be challenging to come to any kind of agreement among stakeholders about what data should be embedded. This goes double for the second and subsequent passes. Moreover, those second and subsequent passes often use data about the document, rather than data in the document. Data a site owner would be unable to provide or even incentivized to falsify. Not less than these problems is the issue of where to run the scorer code. If you're running it locally, you're operationally 90% of the way to the complexity of an inverted index. Why not go all the way?

Then of course there are the economic issues, which, roughly, are: "Why should I pay all this money to host an index of my site that nobody uses when Google will do it for free and charge me nothing?"