Qdrant 1.7.0

> Traditional keyword-based search mechanisms often rely on algorithms like TF-IDF, BM25, or comparable methods. While these techniques internally utilize vectors, they typically involve sparse vector representations. In these methods, the vectors are predominantly filled with zeros, containing a relatively small number of non-zero values. Those sparse vectors are theoretically high dimensional, definitely way higher than the dense vectors used in semantic search. However, since the majority of dimensions are usually zeros, we store them differently and just keep the non-zero dimensions.

Yeah Heap's Law is a bit of a bitch in these situations. Like you'll definitely make sure those vectors are 64 bit if you plan of indexing a proper large number of documents.

I'd also advise caution leaning too much into the vector interpretation of these algorithms, as it's largely viewed as a quaint historical artifact that was bit of a dead end (e.g. as in Croft, Metzler & Strohman 7.2.1)

Just the other day I played with qdrant, using its Python client. Pretty smooth onboarding experience.

I came across two questions. Perhaps some kind folks with more experience can shed some light on these qdrant use cases.

1. for embeddings for use cases such as LLM chat bots, I split internal data into chunks. Those chunks are then vectorized and stored. Alongside the entry itself, I stored the original chunk in metadata. That way, a lookup can immediately feed that into the LLM prompt context, without lookup in a secondary data store by some ID. Feels like a hack. Is that a sensible use case?

2. I resorted to using `fastembed` and generated all embedding client-side. Why is it that qdrant queries, in the ordinary case (also showcased a lot in their docs, e.g. [0]), expect a ready-made vector? I thought the point of vector DBs was to vectorize input data, store it, and later vectorize any text queries themselves?

Having to do all that client-side feels besides the point; for example, what if two separate clients use separate models (I used [1])? Their vectorizations will differ. I thought the DB is the source of truth here.

In any case, fascinating technology. Thanks for putting it together and making it this accessible.

[0]: https://qdrant.tech/documentation/quick-start/#run-a-query

[1]: `sentence-transformers/all-MiniLM-L6-v2`, following https://qdrant.tech/documentation/tutorials/neural-search-fa...

As another signal, check out this report by Streamlit, which shows the popularity of different vector databases among Streamlit apps: https://state-of-llm.streamlit.app/#third

Faiss and Pinecone are at the top (disclosure: I'm from Pinecone). But Faiss isn't really a full-fledged vector DB. Pinecone is a managed option which is out of the question for a company like Twitter and maybe for you (although you should consider it). After that comes Chroma in third, and then Qdrant, and then Weaviate.

Chroma has a big following by virtue of being plugged into the AI ecosystem in SF. Qdrant seems to be doing great work but their location in Europe is probably not helping.

Regarding your last sentence: the European HQ might not exactly help for non-EU customers, but much much more so for actual EU customers (which is a multi billion Dollar market by itself). Sensitive EU companies would not use Pinecone, even if they wanted to.