The Age of PageRank Is Over

Need more money? Economic model not quite working? Companies face this all the time. Some solve it innovatively, survive, and show everyone else the way forward. Others fold or pivot.

But ad revenue short-circuits the hard process of economic innovation by tempting tech companies to take an easy way out, even when the problem is solvable without it.

(All of this wouldn't really be a problem if the ads business model weren't inherently adversarial against users.)

I do think your idea is great and something with a lot of value for many use cases.

The only issue is that it won't surface certain things like restaurant contact info and Wikipedia that people often need to be a top result.

https://kagi.com/search?q=postgres+cte&r=us&sh=5-n8GUySt5qmx...

Even in a hypothetical situation where ads would no longer be the driving force of algorithms, something else will. As soon as something gets large enough, it will be gamed. If not for commercial reasons, it might be cultural/political influence.

I will end by reminding ourselves that us techies should spend some more time with ordinary folks. I agree with everybody here that Google Search has been getting worse and worse for years now, especially for our niche searches.

It's a mistake though to think that this is widely experienced as such. My mum looks up an unknown ingredient in a recipe and within a second sees a picture of it. That means it works. All kinds of personal data might be shared in the process but since you can't really see that, it didn't happen. From her point of view, Google Search works extremely well and is close to magic.

The point being, Google doesn't give a shit that you don't get the best answer for your query on JavaScript closures. Nobody searches for that, and those that do, block ads.

https://trec.nist.gov/

but it takes a huge amount of training/test data to do it and takes a lot of computational work after you've got that data. What TREC revealed is that most of the things that would obviously improve search relevance don't, and it took 5 whole years of 20 teams working on it before a useful discovery was made.

There was an article about TikTok that revealed just how wrongheaded the viewpoint of the current web is. As much as Google fetishizes data, the data collected by sites like YouTube is useless because they offer you too many things to click on so you click around like a chicken with its head cut off. There are maybe 5 things that you like out of 20 that they show you, but which one you click on is random so the signal is mostly noise and worst of all they can't come to the conclusion that you didn't like any of the other 19 things they showed you.

TikTok shows you exactly one thing at a time so the opinion that they capture is meaningful.

It reminds me of one of the first "learning to rank" papers where I talked the management at the CU Library to let the Thorsten Joachims group run our search engine and we realized just how poor of a signal you get from search engine usage and how challenging it is to feed it back to improve your results.

Really marking up judgements for all the results that turn up and using "pooling" to add new results to the original set when your search engine is essential to make really better search. Otherwise you can be just another one of those guys who posts to Medium about the "semantic search" engine he built but can't tell you if the results are any better than any other search engine.