The world needs a non-profit search engine

How many advertisers outside of that country are going to ask YouTube to play their ads in that country in a language different from the dominant language? Probably not many.

Programming reminds me of other professions with high inputs but low per-unit-cost outputs: teaching, music, movies, art, journalism, etc. -- basically anything that is create-once, share-often. All those (for the most part) are things that America's shortsightedly capitalist economy fail to adequately incentivize/reward unless you happen to become a celebrity.

There is already, and will continue to be, a class of human labor output that is intrinsically valuable but which our economy is unable to adequately price. I'd argue that's more an issue with the speculation-driven economy that we have than with the labor in question. It'll only get more drastic we we automate more and more and further amplify human creativity.

It'd be cool to see a nonprofit search engine/email service/office suite/whatever funded in a similar style as NPR, but of course that'd run into political issues at every level.

In an utopian cyberpunk future, what if there were multiple voluntary nonprofit "shadow governments" that you can choose to tithe every month, almost like churches? You can choose between e-governments red, blue, green, yellow, purple, gray, whatever... give them 1% of your income a year in exchange for a suite of services run and staffed by professionals who are salaried but own no equity; they work as a form of civil service, not a wealth-building scheme. Syndicates for the public good, I suppose. Lol, in all the video games, these usually turn into private military companies with killer androids, but what if they just, uh, provided really good email (and automatic online driver's license renewals) instead?

Such systems would probably never be able to attract the best talent (unless they turn into something like Mozilla, which is a big enterprise masquerading as a nonprofit), but you often don't NEED the best. Wikipedia, the sum total of human knowledge, is also the sum total of human mediocrity (with plenty of redundant efforts, infighting, unoptimized problems, etc.). But having stable collaborative communities is something that is critical for producing works of intrinsic value, and developing slow-trickle funding streams for such communities -- the kind that can sustain without turning them into potential get-rich-quick schemes, or subject them to violent boom-bust cycles -- is what allows them to both keep working effectively, AND keeps away the exploitative get-rich-quick types looking to subvert public labor for personal gain.

We need a funding model that provides enough income to attract people who want to do it for the public good, but not so much that it also attracts people who want to turn it into personal wealth. In high-input, low-per-unit-cost services, optimizing for maximum (as opposed to sufficient) monetization will too often mean that the users themselves become the product, as we see time and again with Google, Facebook, Twitter, and basically the entire modern online economy. It's the difference between Amazon and your local library, Oracle and Postgres, EvilOS and Linux, etc.

Things don't have to be artificially scarce if we can learn to ask, "How can I make this as widely accessible as possible while ensuring I have my own basics needs covered?" instead of "How can I make this as monetarily valuable as possible?"

There are always going to be people who want to just do things to make the world a better place, just as surely as there are always going to be people who want to optimize for personal profit. We have every system to fund the latter, but not so much the former right now. Is the for-profit business model the only, or best, way?

That is, even if a site wanted to, there's no way for it to declare "I have content related to X". Even better would be if these indices could then be distributed in a cache-and-forward model similar to how DNS (another distributed discovery index) works. There was some exceedingly rudimentary attempt at this through elements such as keyword meta tags, but even at best these referenced a vanishingly small fraction of the actual content of a site or article. Sitemaps also address a component of the problem, but again, only in part.

Some might see a few immediate issues. One is that not all site are sufficiently dynamic to know what content they actually contain. To an extent this might be addressable through extension to the webserver protocol such that a server would be aware, or become aware, of what content it contained.

Another is that a site might in some instances be inclined to misrepresent what it contained. This may be hard for some to believe, but I'm given to understand it occasionally does occur. To help guard against this, there might be vetted indices, in which one or more third parties vouch for the validity of an index. These reputation-sources could of course themselves be assessed for accuracy.

But if sites were responsible for reporting on what content they actually contained, and could be constrained to doing so accurately, a huge part of the overhead in creating independent search engine, and breaking the seach-engine monopoly, would be eliminated.

One might imagine why certain existing gatekeepers over Web standards might oppose such an initiative.

There would still remain other problems to solve within search space. It's possible to divide General Web Search into a set of specific problems:

- Site crawling: this includes determining search targets, any exclusions from such lists, and performing the actual crawling. Self-indexing addresses part of this problem.

- Indexing: Mapping of actual contents to keyword and query terms which might address that content.

- Ranking: Assigning a preference / deprecation to specific sites. This is essentially a trust / reputation assessment, with a canonicity / authenticity assessment (e.g., where did a specific item or document first appear).

- SEO: This is the Red Queen's Race issue in addressing insincere / malicous actors. Strong and durable penalties for abuse, and long-term reputational accrual, should be useful here.

- Query interpretation: There's a considerable art to figuring out what a question actually means. In some cases queries should be taken strictly verbatim. Quite often, however, interpretation is necessary. How those alternatives are posed might vary, with an option not often employed presently being to suggest a range of potential interpretations or related queries which might produce better results for specific query scenarios.

- Presentation: This is generation of the serch engine result page itself, incorporating several of the other considerations listed, but also addressing usability, accessibility, clarity, and other concerns.

- Revalidation: As the editors of the Hitchiker's Guide observed, the Universe is not static, and circumstances change. Revalidating, revisiting, and revising results and reputational assessments is necessary.

- Monetisation/Funding: I'm partial to a public goods model, or perhaps a farebox role via ISPs, pro-rated to general income/wealth within a region. Advertising, as a famous Stanford research paper prophetically observed, forces disallignment with searchers' interests and objectives.