Everyone I talk to who is building some vector db based thing sooner or later realizes they also care about the features of a full-text search engine. They care about filtering, they care to some degree about direct lexical matches, they care about paging, getting groups / facet counts, etc. Vectors, IMO, are just one feature that a regular search engine should have. IMO currently Vespa does the best job of this, tho…
Until very recently, “dense retrieval” was not even as good as bm25, and still is not always better. I think a lot of people use dense retrieval in applications where sparse retrieval is still adequate and much more flexible, because it has the hype behind it. Hybrid approaches also exist and can help balance the strengths and weaknesses of each. Vectors can also work in other tasks, but largely people seem to be usi…
I don't think there will ever be an answer to what is the best way of doing information retrieval for a search engine scale corpus of document that is superior for every type of queries.