I feel like most of the points raised in the article are solved by “use pgvector”, and then I’m very skeptical of handing over responsibility for API calls for creating the embeddings to the DB itself? I already have a software layer that knows how to do things like logs, API call failures? Having the DB handle fetching data from external sources feels like the wrong abstraction to me.
Vector databases are the wrong abstraction
71–80 of 95 posts
Re: Vector databases are the wrong abstraction
#72Clever! A method that has worked well for me: divorced databases. The first database is a plaintext database that stores rows: id, data, and metadata and the second database is a vector database that stores id, embedding. whenever a new row is added the first database makes a POST request to the second database. The second database embeds the data and returns the id of its row. The first database uses that ID to stor…
It sounds like this is pretty similar to the approach that the post is advocating against although I can see your reasoning behind this.
Re: Vector databases are the wrong abstraction
#73Earlier quoted context omitted.
Chroma certainly doesn't have the most advanced API in this area, but you can for sure store chunks or documents, its up to you. If your document size is too large to generate embeddings in a single forward pass, then yes you do need to chunk in that scenario. Oftentimes though, even if the document does fit, you choose to chunk anyways or further transform the data with abstractive/extractive summarization technique…
(Post co author) We absolutely agree that chunking is critical for good RAG. What I think you missed in our post is that the vectorizer allows you to configure a chunking strategy of your choice. So you store the full doc but then the system well chunk and embed it for you. We don’t blindly embed the full document.
I didn't miss that detail, I just don't think chunking alone is where the complexity lies and that the pgai feature set isn't really differentiated at all from other offerings in that context. My commentary about full documents was responding directly to your comment here in this thread more so than I was the article (you claimed chroma can only insert chunks, which isn't accurate, and I expanded from there).
Re: Vector databases are the wrong abstraction
#74I feel like most of the points raised in the article are solved by “use pgvector”, and then I’m very skeptical of handing over responsibility for API calls for creating the embeddings to the DB itself? I already have a software layer that knows how to do things like logs, API call failures? Having the DB handle fetching data from external sources feels like the wrong abstraction to me.
I agree. However, I think what they are saying is the embedding should just be like any other index. I mean, yeah it should be but that isn't reality. There are massive latencies involved as well as costs.
Perhaps in ~10 years embedding / chunking approaches will be so mature that there will just be one way to do it and will take no more time than updating a btree but that certainly isn't the case now.
I think the right abstraction for today would be for OpenAI to manage the vector search. It is kind of weird to send all of the data to a service only to have it compute a vector and hand it back to me. I have to figure out how to chunk it etc (I'm sure they would do a better job than I would). I should just have to deal with text ideally. Someone else can figure out how to return the best results.
Re: Vector databases are the wrong abstraction
#75Wow, actually a good point I haven't seen anyone make. Taking raw embeddings and then storing them into vector databases, would be like if you took raw n-grams of your text and put them into a database for search. Storing documents makes much more sense.
This is how most modern vector dbs work, you usually can store much more than just the raw embeddings (full text, metadata fields, secondary/named vectors, geospatial data, relational fields, etc).
Re: Vector databases are the wrong abstraction
#76I agree with the author - introducing a vector database often isn't worth the extra complexity. Personally, I can vouch for ParadeDB: https://www.paradedb.com/ It adds extra extensions to PostgreSQL which enable vector indexing, full text search and BM25. Works great and developers are helpful! The major difference is that you must generate the embeddings by yourself, but I consider it an upside - to each their own :…
We could add support for something like `pg_vectorize` in order to generate embeddings directly from the database. We simply haven't seen enough demand yet. Perhaps we haven't listened hard enough :')
Re: Vector databases are the wrong abstraction
#77Earlier quoted context omitted.
I made something similar, but used duckDB as the vector store (and query engine)! It’s impressively fast https://github.com/patricktrainer/duckdb-embedding-search
Amy specific reason to use dDB? I've got a crapload of json q & a formatted discussions on a topic, and am trying to figure out if I just store it somewhere and query it, or do I also do vector embeddings, kinda lost with all the possible options.
Embeddings are what encode the “meaning” of a given text. Similarity search works by computing the angle between your query vector and the rest of the vectors already stored.
DuckDB (and columnar stores in general) is great at aggregation. It’s particularly well suited because DuckDB is a single file. There’s no server to muck with.
Re: Vector databases are the wrong abstraction
#78Earlier quoted context omitted.
Post co-author here. Really appreciate the feedback. Your point about HNSW being resource intensive is one we've heard. Our team actually built another extension called pgvectorscale [1] which helps scale vector search on Postgres with a new index type (StreamingDiskANN). It has BQ out the box and can also store vectors on disk vs only in memory. Another practice I've seen work well is for teams use to use a read rep…
Thanks for your answer. I hear you on using a read-replica to serve embedding-based queries, but I worry there are lots of cases where that breaks down in practice: presumably you still need to do a bunch of IO on the primary to support insertion, and presumably reconstituting an index (e.g. to test out new hyperparameters) isn't cheap; at least you can offload the memory requirements of reading big chunks of your gr…
So… maybe 15 or 20 years ago I had setup MySQL servers such that some replicas had different indexes. MySQL only had what we would now call logical replication.
So after setting up replication and getting it going, I would alter the tables to add indexes useful for special purposes including full text which I did not being built on the master or other replicas.
I imagine, but can not confirm, that you could do something similar with PostgreSQL today.
Re: Vector databases are the wrong abstraction
#79I feel like most of the points raised in the article are solved by “use pgvector”, and then I’m very skeptical of handing over responsibility for API calls for creating the embeddings to the DB itself? I already have a software layer that knows how to do things like logs, API call failures? Having the DB handle fetching data from external sources feels like the wrong abstraction to me.
I agree. However, I think what they are saying is the embedding should just be like any other index. I mean, yeah it should be but that isn't reality. There are massive latencies involved as well as costs. Perhaps in ~10 years embedding / chunking approaches will be so mature that there will just be one way to do it and will take no more time than updating a btree but that certainly isn't the case now. I think the ri…
> I think the right abstraction for today would be for OpenAI to manage the vector search
So I disagree, but they have a very easy-to-use RAG system in beta that does what you want.
In my use cases, fine-grained control over chunking and so on is application-level code. I’m using an LLM to split documents into subdocuments with context (and location) and then searching those subdocuments, while pushing the user to the source
Re: Vector databases are the wrong abstraction
#80At my current company, we used Postgres with pgvector so the text is co-located with the embeddings on the same rows. At first, I was a bit apprehensive about the idea of getting so close to the nitty-gritty technical details of computing vector embeddings and doing cosine similarity matching but actually it has been wonderful. There is something magical about working directly with embeddings. Computing, serializing and storing everything yourself is actually surprisingly simple. Don't let the magic scare you.
Recently I've been doing hardcore stuff like taking an old hierarchical clustering library and substituting the vector distance functions with a cosine similarity function so that it groups/clusters records based on similarity of their embeddings. It's funny reading the README of that 10 year old library and they're showing how to use it to do tedious stuff like grouping together 3-dimensional color vectors. I'm using it to cluster together content based on meaning similarity using vectors of over 1.5k dimensions. Somehow, I don't think the library authors saw that coming.
How great is it to come across a library which hasn't been updated in 10 years and yet is flexible and simple enough that it can be re-purposed to serve a radically more advanced use case which would have been beyond the author's imagination at the time...
I think the most surprising aspect about the whole experience is that working with the embeddings directly makes it feel like your database is intelligent; but you know it's just a plain old dumb database and all the embeddings were pre-computed.