Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Vector databases are the wrong abstraction

timescale.com

Post reply on HN
Print view View original

41–50 of 95 posts

Re: Vector databases are the wrong abstraction

#41

Post by avthar »

whakim wrote: post #40

This is actually really cool, and despite what I'm sure will come off as (constructive) criticism, I am very impressed! First, I think you oversell the overhead of keeping data in sync and the costs of not doing so in a timely manner. Almost any distributed system that is using multiple databases already needs to have a strategy for dealing with inconsistent data. As far as this problem goes, inconsistent embeddings…

Post co-author here. Really appreciate the feedback.

Your point about HNSW being resource intensive is one we've heard. Our team actually built another extension called pgvectorscale [1] which helps scale vector search on Postgres with a new index type (StreamingDiskANN). It has BQ out the box and can also store vectors on disk vs only in memory.

Another practice I've seen work well is for teams use to use a read replica to service application queries and reduce load on the primary database.

To answer your third question, if you combine Pgai Vectorizer with pgvectorscale, the limitations around filtered search in pgvector HNSW are actually no longer present. Pgvectorscale implements streaming filtering, ensuring more accurate filtered search with Postgres. See [2] for details.

[1]: https://github.com/timescale/pgvectorscale [2]: https://www.timescale.com/blog/how-we-made-postgresql-as-fas...

Re: Vector databases are the wrong abstraction

#42

Post by metalwhale »

motoxpro wrote: post #37
metalwhale wrote: post #36

Earlier quoted context omitted.

Thank you for sharing this! I have one question: Is there any plan to add support for local LLM / embeddings models?

"Right now the system only supports OpenAI as an embedding provider, but we plan to extend with local and OSS model support soon." In the post you responded to

Haha I feel so dumb now. Thank you!

Re: Vector databases are the wrong abstraction

#43

Post by wild_egg »

ekianjo wrote: post #34
pjot wrote: post #15

Earlier quoted context omitted.

I made something similar, but used duckDB as the vector store (and query engine)! It’s impressively fast https://github.com/patricktrainer/duckdb-embedding-search

There is vector type data available in duckdb now?

They call it a fixed size array type but, yes. It was added earlier this year. Works really great

https://duckdb.org/2024/05/03/vector-similarity-search-vss.h...

Re: Vector databases are the wrong abstraction

#44

Post by spmurrayzzz »

jeffchuber wrote: post #30
avthar wrote: post #28

Earlier quoted context omitted.

Post co-author here. The point is a little nuanced, so let me explain: You are correct in saying that that you can store embeddings and source data together in many vectordbs. We actually point this out in the post. The main point is that they are not linked but merely stored alongside each other. If one changes, the other one does not automatically change, making the relationship between the two stale. The idea behi…

i know it is the case in chroma this is supported out of the box with 0 lines of code. i’m pretty sure it’s supported everywhere else in no more than 3 lines of code.

This is also the case with weaviate (as you assumed). If you update the value of any previously vectorized property, weaviate generates new vectors automatically for you.

Re: Vector databases are the wrong abstraction

#45

Post by pjot »

ekianjo wrote: post #34
pjot wrote: post #15

Earlier quoted context omitted.

I made something similar, but used duckDB as the vector store (and query engine)! It’s impressively fast https://github.com/patricktrainer/duckdb-embedding-search

There is vector type data available in duckdb now?

Yep! It was added in v0.10.0 - which was released a month or two after I made this.

This is using v0.9.1

Re: Vector databases are the wrong abstraction

#46

Post by spmurrayzzz »

cevian wrote: post #31
jeffchuber wrote: post #30

Earlier quoted context omitted.

i know it is the case in chroma this is supported out of the box with 0 lines of code. i’m pretty sure it’s supported everywhere else in no more than 3 lines of code.

as far as I can tell Chroma can only store chunks, not the original documents. This is from your docs `If the documents are too large to embed using the chosen embedding function, an exception will be raised`. In addition it seems that embeddings happen at ingest time. So, if, for example, the OpenAI endpoint is down the insert will fail. That, in turn means your users need to use a retry mechanism and a queuing syst…

Chroma certainly doesn't have the most advanced API in this area, but you can for sure store chunks or documents, its up to you. If your document size is too large to generate embeddings in a single forward pass, then yes you do need to chunk in that scenario.

Oftentimes though, even if the document does fit, you choose to chunk anyways or further transform the data with abstractive/extractive summarization techniques to improve your search dynamics. This is why I'm not sure the complexity noted in the article is relevant in anything beyond a "naive RAG" stack. How its stored or linked is an issue to some degree, but the greater more complex smell is in what happens before you even get to that point of inserting the data.

For more production-grade RAG, just blindly inserting embeddings wholesale for full documents is rarely going to get you great results (this varies a lot between document sizes and domains). So as a result, you're almost always going to be doing ahead-of-time chunking (or summarization/NER/etc) not because you have to due to document size, but because your search performance demands it. Frequently this involves more than one embeddings model for capturing different semantics or supporting different tasks, not to mention reranking after the initial sweep.

That's the complexity that I think is worth tackling in a paid product offering, but the current state of the module described in the article isn't really competitive with the rest of the field in that respect IMHO.

Re: Vector databases are the wrong abstraction

#47

Post by dmezzetti »

I've been in the vector database space for a while (primary author of txtai). I do think vector indexing in traditional databases with tools like pgvector is a good option.

txtai has long had SQLite + Faiss support to enable metadata filtering with vector search. That pattern can take you farther than you think.

The design decisions I've made is to make it easy to plug different backends in for metadata and vectors. For example, txtai supports storing both in Postgres (w/ pgvector). It also supports sqlite-vec and DuckDB.

I'm not sure there is a one-size-fits-all approach. Flexibility and options seems like a win to me. Different situations warrant different solutions.

Re: Vector databases are the wrong abstraction

#48

Post by mind-blight »

avthar wrote: post #2

Hey HN! Post co-author here, excited to share our new open-source PostgreSQL tool that re-imagines vector embeddings as database indexes. It's not literally an index but it functions like one to update embeddings as source data gets added, deleted or changed. Right now the system only supports OpenAI as an embedding provider, but we plan to extend with local and OSS model support soon. Eager to hear your feedback and…

This is super cool! One suggestion for the blog: I would put "re-imagines vector embeddings as database indexes. It's not literally an index but it functions like one to update embeddings as source data gets added, deleted or changed." as a tl/dr at the top.

It wasn't clear to me why this was significantly different than using pg_vector until I read that. That makes the rest of the post (e.g. why this you need the custom methods in a `SELECT`) make a lot more sense in context

Re: Vector databases are the wrong abstraction

#49

Post by whakim »

avthar wrote: post #41
whakim wrote: post #40

This is actually really cool, and despite what I'm sure will come off as (constructive) criticism, I am very impressed! First, I think you oversell the overhead of keeping data in sync and the costs of not doing so in a timely manner. Almost any distributed system that is using multiple databases already needs to have a strategy for dealing with inconsistent data. As far as this problem goes, inconsistent embeddings…

Post co-author here. Really appreciate the feedback. Your point about HNSW being resource intensive is one we've heard. Our team actually built another extension called pgvectorscale [1] which helps scale vector search on Postgres with a new index type (StreamingDiskANN). It has BQ out the box and can also store vectors on disk vs only in memory. Another practice I've seen work well is for teams use to use a read rep…

Thanks for your answer. I hear you on using a read-replica to serve embedding-based queries, but I worry there are lots of cases where that breaks down in practice: presumably you still need to do a bunch of IO on the primary to support insertion, and presumably reconstituting an index (e.g. to test out new hyperparameters) isn't cheap; at least you can offload the memory requirements of reading big chunks of your graph into memory onto the follower though.

Cool to see the pgvectorscale stuff; it sounds like the approach for filtering is not dissimilar to the direction that the pgvector team are taking with 0.8.0, although the much-denser graph (relative to HNSW) may mean the approach works even better in practice?

Re: Vector databases are the wrong abstraction

#50

Post by TechDebtDevin »

avthar wrote: post #2

Hey HN! Post co-author here, excited to share our new open-source PostgreSQL tool that re-imagines vector embeddings as database indexes. It's not literally an index but it functions like one to update embeddings as source data gets added, deleted or changed. Right now the system only supports OpenAI as an embedding provider, but we plan to extend with local and OSS model support soon. Eager to hear your feedback and…

I'm doing something similar with go + postgres
Post reply on HN