Ask HN: Is 'search' a solved problem?

If you squint a bit, you will find every computation is a kind of search. You are searching for the "output" based on the given "input". Ex: Deep learning is just searching for the weights. What people usually called search is where the search space is explicitly defined (a set of records in db etc) but in most cases the search space is there just that it is implicitly defined in the computation problem description.

I see opportunity for niche search engines, there are several areas that google does not do well in on purpose it seems.

I think the truly hard part is that so many people accept whatever default is already there. If they get an android phone - they use the search box there. If they are using chrome browser, whatever input box is there on the first screen is obviously the url bar and use that (you and I may know the difference, the average user doesn't care, it's one less click to just type 'google' into the url box in the center of the page, of fbook or whatever, then google brings up the url you were going to (not searching).

This is why I think there is much less hype about competing in this space. Unless there is a thing forcing companies to put other browsers and search boxes on phones, tablets and chromebooks like the microsoft IE debacle so long ago.. then trying to be the next google is impossible, even if you had better results, better tech, etc.

Regardless of that, I think it's quite possible to make much better niche search engines and get them used. If ten micro engines could make 1% of googles revenues each, that would be a decent amount of money in my neck of the woods.

I'd like to see other people post more sources about search tech in general, several searches last year only brought a few info bits on what it may cost to create an index of the net - someone posted some numbers using servers bought off ebay and a rack at hurricane I think - had some numbers for the cost of servers to pull a new index every month or so?

Certainly the tech and costs have changed since that was published, but not much I've seen.

I'm pretty excited at this project posted recently: https://news.ycombinator.com/item?id=16976941 ( Show HN: A search engine that doesn't track you, where users vote for results (github.com) )

I am hoping to get some people together to make a less persnickety and fussbudgety search option for people who don't want to be babysit with censoring kids gloves when looking for fun things.

If anyone wants to make a couple adults only engines, or ones that are more fun, let me know.

Average people talk in slang and cut up about less high brow things, the big G gives rank to the college papers and deranks for so many things, it's on the road to being the next yellow pages and sciences journal, but not the place to go when you want fun things anymore.

Search means a lot of things, but even if we limit to mean web-search, as most people understand it, there is a lot more to it than the actual technology that matches queries to documents.

IR is for all intents and purposes a solved problem -- in fact it was solved a long time ago, and I highly recommend the seminal book “Managing Gigabytes”. I also recommend https://github.com/phaistos-networks/Trinity/wiki/IR-Search-... this page(disclaimer: I am maintaining it) for some interesting/important links to IRC technologies, developments, etc. While some novel ideas come out from time to time, the fundamentals haven’t changed -- progress there is incremental and mostly specific to different encoding schemes or ways to execute queries faster by using JIT or more cache-aware datastructures, etc.

Managing and queries documents based on keywords and boolean operators is one thing, and Lucene/Solr, and Trinity (https://github.com/phaistos-networks/Trinity) among other technologies can be used to take care of those challenges. But that’s the easy part (assuming you can do this fast enough, because you almost always can’t afford long-running queries):

- User Interfaces: Not just how results are presented, but also how users can construct or input queries. What options can be come available for filtering matches? - Ranking: Precision is key, and rather simple formulas (tf/idf, BM25, etc) generally don’t work well for many/most domains. Furthermore, ranking is almost always not just about relevancy. It factors in static context scores (e.g document “popularity”), personalisation biases(how likely is it for user to mean Soccer or American Football for [football]),and other signals, fused together somehow to determine the final ranking of matched documents. - Scale: Getting everything right is one thing, getting everything right at massive scale is whole different game. What may work on small scale(algorithms, technologies, services) may not work at all when you scale out. - Everything else not directly related to search but either important or fundamental to a good experience/business: from matching queries to ads, to analytics, to autosuggestions, to training ML models to power all that, etc.

Web search is not a zero sum game. Bing makes over 3nb / year and while it may not have a chance to catch up with Google anytime soon, that’s a great business right there. Ditto for DDG. There are also companies that offer a different or better experience and access to datasets google doesn’t yet.

So, all told, search may be solved only in terms of the basic IR technology that makes it all work, and arguably a lot better than it used to be in terms of user interfaces, ranking, etc, but it will take a lot longer until those other aspects of web search may be considered ‘solved’.