We need it - I’d use 3 search engines. It’s not like MMOs or social media where 1 winner takes all. I think people want more search options, cloud, email, and so-on.
I once tried to index the web with local Mistral 7b. Here’s what I did:
1. Copy/pasted an English dictionary and removed the definitions, so I just had an Array of about 500k words.
2. Wrote a simple loop to iterate over the words, with the instruction to Mistral to try to guess 10 URLs related to that word. I would typically get 2-3 sites per word.
3. When visiting a site, also check that site for links to other URLs, and summarize it with hash tags, visiting the new sites as well and recursively doing that.
4. Keep a tally of how many sites are being linked to for the page’s “rank”.
In the end you should have a massive list of objects with valid URLs ranked by how popular they are (to each other, by link), and a set of searchable hash tags for each.
I ran this for days and I never got past B, and almost every site was adult content. Most of the sites came from other sites. But it does work! Janky page crawling and ranking.
Anyway more people should take on web search, I agree. LLMs offer shortcuts.