91–95 of 95 posts
I probably won't use this right now when my app is so small because it of the complexity managing another service introduces. But I imagine as it gets bigger this would make things simpler.
Clever! A method that has worked well for me: divorced databases. The first database is a plaintext database that stores rows: id, data, and metadata and the second database is a vector database that stores id, embedding. whenever a new row is added the first database makes a POST request to the second database. The second database embeds the data and returns the id of its row. The first database uses that ID to stor…
I agree with the author - introducing a vector database often isn't worth the extra complexity. Personally, I can vouch for ParadeDB: https://www.paradedb.com/ It adds extra extensions to PostgreSQL which enable vector indexing, full text search and BM25. Works great and developers are helpful! The major difference is that you must generate the embeddings by yourself, but I consider it an upside - to each their own :…
> I consider it an upside I'm curious why you consider an upside. Hypothetically speaking, wouldn't it be better if the embeddings could automatically be updated when you want them to be? Is the problem that it's not easy to automated based on the specific rules of when you want updates to happen?
- What if certain rows in a table don't need to be embedded?
- What if we use a single API key for embedding database rows and user queries and it hits a rate limit - how to prioritize user queries?
- What if some rows should be vectorized using a different model, depending on an external configuration?
Great point! (Disclaimer: I work for Elastic) Elasticsearch has recently added a data type called semantic_text, which automatically chunks text, calculates embeddings, and stores the chunks with sensible defaults. Queries are similarly simplified, where vectors are calculated and compared internally, which makes a lot less I/O and a lot simpler client code. https://www.elastic.co/search-labs/blog/semantic-search-sim…
I made something similar, but used duckDB as the vector store (and query engine)! It’s impressively fast https://github.com/patricktrainer/duckdb-embedding-search
DuckDB has two configurable options for concurrency:
1. One process can both read and write to the database.
2. Multiple processes can read from the database, but no processes can write (access_mode = 'READ_ONLY').