"An index (yellow pages) of what's available would help me search for a product."
This is what "tech" companies are seeking to prevent. If the user has no idea of what's available, then the "tech" company can "suggest" its advertisers' products and services. "Recommendations" are supposed to be technologically indistinguishable from "search". Gaslighting indeed.
But don't believe me. Get it from the horse's mouth. Read Google's brief in opposition in the Gonzalez case.
https://www.supremecourt.gov/DocketPDF/21/21-1333/229391/202...
Google wants the US Supreme Court to believe that recommendations systems and search engines are the same thing.1 If the justices were born into the world of Google, Facebook and now TikTok, that argument might work. But a majority were born before the www existed. The difference between search and recommendations is obvious and significant.
"Tech" companies, which produce no content themselves, want to recommend whatever gets "eyeballs", so they can keep the lights on by selling online advertising services. Popularity is the all-important metric. They want to run websites that serve hundreds of millions of people, relegating the rest of the web's sites to brief tangents, temporary distractions. Why. Because those conditions are optimal for advertising. For a variety of reasons, they are unable to satisfactorily filter, curate, censor, etc. the content they collect from others.
This only works as a lucrative racket so long as the "tech" company cannot be sued for the content they take from others and use as bait to lure in users to be targeted with advertising. Generally, the preaviling interpretation of Section 230 has effectively prevented such litigation.
If that interpretation changes, it could spell the end of this enormous racket. Big Tech would not be able to continue to profit from user-generated content by steering web users toward what is popular and creating large audiences attractive to advertisers. Smaller websites that produce their own content could again become relevant.
In the past, I have had personal projects where I create an alternative way to explore the www, using a Yellow Pages type of process. IMO, the reason no one seems to care about this area anymore2 is not that such projects are infeasible or too challenging, it is that there is no financial incentive to pursue them. Big Tech sold out the web. Big Tech has sought to portray the web as a 100% commercial network. It has thus become a sewer of low quality "content" and try-hard advertising, while the data collection, tracking and ad serving apparatus purchased with all that wasted ad spend is state of the art.
1. Because there is precedent that suggests 230 protects search engines. There is however no precedent that suggests it protects "recommendations".
2. I was recently looking at the original sources for WAIS on a simtel mirror. All the pieces were there. The www had so much potential. At least Brewster Kahle went on to do something useful, the Internet Archive. I still sometimes use z39.50 today to do searches on library databases. To me it represents a time before advertising infected computer networks and eventually the internet.