Hmm, I have been thinking about this more lately.
How to get quality results, and a sustainable, community-led search engine?
=== Contexts ===
A "search engine" such as Google is good at many things, and extremely bad at others. The main issue with it in my view is that it lacks context about what you are looking for. The main context you can ask for is "Videos", "Pictures", etc.
* Specifying context takes time, so it's OK for long searchs Google sucks at (find this specific article I read a while back). Take your time while you specify language, exact/fuzzy match, publication date, background color, author name or any number of things you know about your search.
* Some requests can be processed with instant answers, that's good news as it fits the open source model quite well.
* Lastly, the other requests. Some are asked like a question and might require NLP to sort trough. Quite hard IMO, it might get better but will still require compute power if done server-side. It's mostly: parse the question to find the context, and perform a contextual keyword search/instant answer.
* And those that aren't questions: "regular", keyword-based requests, that "just" require a big index and a big infrastructure to search it.
=== Hardware ===
Now, we are left with the cost centers: hardware. IMO, the only way to scale is to rely on the community and distribute things.
* Databases: if this is a community project, and not too latency-sensitive, the community can help by distributing them over a p2p network, even with a single source of trust.
* Queries, walking the database: delegating processing to untrusted third-parties is a bit more dangerous. Maybe allow each user to specify a list of trusted servers? Can be centralized and clients ask the network, though it might leak part of their search, depending on the index method. Could be client-side?
* Processing the answers: client-side, or trough any number of frontends (like searx).
* Crawlers: crawling the net isn't cheap. You could use one or multiple sources of trust. Domain-specific crawlers, like hinted at in Drew's post. Maybe crawl on demand or trough the user's computer (web extension that indexes as the user browses, and allows them to full-text search their history; share it or not).
=== Content ===
For some measure of quality, I find that websites that do not have advertisements offer better-quality content. That's likely due to conflicting interests. It would be great if the semantic web mandated disclosing revenue sources. You could downrank or avoid crawling sites with ads and/or Google Analytics, for instance. This could be abused to an extent if the service ever becomes popular, but heh https://xkcd.com/810/
Domain-specific crawlers would be nice as well.
=== Added value ===
To be adopted, the service needs to be better than the original in some ways. I think that a new search engine should not try to conquer the masses at first. Instead, find some people that are not satisfied with the current offering and court them. Currently, I think this isn't met by advanced search: exclude websites protected by recaptcha, only include websites that are less than X years old, no ads, etc.
Allow users to create their own contexts and easily switch them: bangs, tabs, date/time/geoip, etc. Have them create contexts dedicated to their activities: programming is an obvious one, but so is cooking, gardening, encyclopedic search, language usage/dictionaries, etc.
=== Monetization ===
At that point, I am not sure it can ever turn a profit? EU grants? Consulting? Help webmaster set up search on their own website? Sell desktop indexing software?
Well, I do have some ideas around content curation, but I am a tad reticent to share them here, and not sure they are more useful than the above.