Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Vector databases are the wrong abstraction

timescale.com

Post reply on HN
Print view View original

31–40 of 95 posts

Re: Vector databases are the wrong abstraction

#31

Post by cevian »

jeffchuber wrote: post #30
avthar wrote: post #28

Earlier quoted context omitted.

Post co-author here. The point is a little nuanced, so let me explain: You are correct in saying that that you can store embeddings and source data together in many vectordbs. We actually point this out in the post. The main point is that they are not linked but merely stored alongside each other. If one changes, the other one does not automatically change, making the relationship between the two stale. The idea behi…

i know it is the case in chroma this is supported out of the box with 0 lines of code. i’m pretty sure it’s supported everywhere else in no more than 3 lines of code.

as far as I can tell Chroma can only store chunks, not the original documents. This is from your docs `If the documents are too large to embed using the chosen embedding function, an exception will be raised`.

In addition it seems that embeddings happen at ingest time. So, if, for example, the OpenAI endpoint is down the insert will fail. That, in turn means your users need to use a retry mechanism and a queuing system. All the complexity we describe in our blog.

Obviously, I am not an expert in Chroma. So apologies in advance if I got anything wrong. Just trying to get to the heart of the differences between the two systems.

Re: Vector databases are the wrong abstraction

#32

Post by cevian »

bryantwolf wrote: post #24

Hey, this looks great! I'm a huge fan of vectors in Postgres or wherever your data lives, and this seems like a great abstraction. When I write a sql query that includes a vector search and some piece of logic, like: ``` select name from users where age > 21 order by limit 10; ``` Does it filter by age first or second? I've liked the DX of pg_vector, but they do vector search, followed by filtering. It seems like tha…

(post co-author here)

It could do either depending on on what the planner decides. In pgvector it usually does post-filtering in practice (filter after vector search).

pgvector HNSW has the problem that there is a cutoff of retrieving some constant C results and if none of them match the filter than it won't find results. I believe newer version of pgvector address that. Also pgvectorscale's StreamingDiskANN[1] doesn't have that problem to begin with.

[1]: https://www.timescale.com/blog/how-we-made-postgresql-as-fas...

Re: Vector databases are the wrong abstraction

#33

Post by brewii »

jeffchuber wrote: post #25
bryantwolf wrote: post #24

Hey, this looks great! I'm a huge fan of vectors in Postgres or wherever your data lives, and this seems like a great abstraction. When I write a sql query that includes a vector search and some piece of logic, like: ``` select name from users where age > 21 order by limit 10; ``` Does it filter by age first or second? I've liked the DX of pg_vector, but they do vector search, followed by filtering. It seems like tha…

pg_vector does post-filtering, not pre-filtering

timescaledbs pg_vector_scale extension does pre-filtering thankfully. shame i cant get it in RDS though

Re: Vector databases are the wrong abstraction

#34

Post by ekianjo »

pjot wrote: post #15
morgango wrote: post #6

Great point! (Disclaimer: I work for Elastic) Elasticsearch has recently added a data type called semantic_text, which automatically chunks text, calculates embeddings, and stores the chunks with sensible defaults. Queries are similarly simplified, where vectors are calculated and compared internally, which makes a lot less I/O and a lot simpler client code. https://www.elastic.co/search-labs/blog/semantic-search-sim…

I made something similar, but used duckDB as the vector store (and query engine)! It’s impressively fast https://github.com/patricktrainer/duckdb-embedding-search

There is vector type data available in duckdb now?

Re: Vector databases are the wrong abstraction

#35

Post by ekianjo »

3abiton wrote: post #23
binarymax wrote: post #16

Earlier quoted context omitted.

It’s impossible to answer that question without knowing what content/query domain you are embedding. Checkout MTEB leaderboard, dig into the retrieval benchmark, and look for analogous datasets.

So we're talking maximizing embedding model per use case? Medical dats would require differnet model than say sales data? Sounds very fragmented approach.

The answer lies with a validation dataset that you create for testing.

Re: Vector databases are the wrong abstraction

#36

Post by metalwhale »

avthar wrote: post #2

Hey HN! Post co-author here, excited to share our new open-source PostgreSQL tool that re-imagines vector embeddings as database indexes. It's not literally an index but it functions like one to update embeddings as source data gets added, deleted or changed. Right now the system only supports OpenAI as an embedding provider, but we plan to extend with local and OSS model support soon. Eager to hear your feedback and…

Thank you for sharing this! I have one question: Is there any plan to add support for local LLM / embeddings models?

Re: Vector databases are the wrong abstraction

#37

Post by motoxpro »

metalwhale wrote: post #36
avthar wrote: post #2

Hey HN! Post co-author here, excited to share our new open-source PostgreSQL tool that re-imagines vector embeddings as database indexes. It's not literally an index but it functions like one to update embeddings as source data gets added, deleted or changed. Right now the system only supports OpenAI as an embedding provider, but we plan to extend with local and OSS model support soon. Eager to hear your feedback and…

Thank you for sharing this! I have one question: Is there any plan to add support for local LLM / embeddings models?

"Right now the system only supports OpenAI as an embedding provider, but we plan to extend with local and OSS model support soon."

In the post you responded to

Re: Vector databases are the wrong abstraction

#38

Post by hhdhdbdb »

cevian wrote: post #18
hhdhdbdb wrote: post #11

Earlier quoted context omitted.

Pretty smart. Why is the DB api the abstraction layer though? Why not two columns and a microservice. I assume you are making async calls to get the embeddings? I say that because it seems n unsual. Index would suit sync better. But async things like embeddings, geo for an address, is this email considered a spammer etc. feel like app level stuff.

(post co-author here) The DB is the right layer from a interface point of view -- because that's where the data properties should be defined. We also use the DB for bookkeeping what needs to be done because we can leverage transactions and triggers to make sure we never miss any data. From an implementation point of view, the actual embedding does happen outside the database in a python worker or cloud functions. Mer…

That is arguable because while it is a calculated field, it is not a pure one (IO is required), and not necessarily idempotent, not atomic and not guaranteed to succeed.

It is certainly convenient for the end user, but it hides things. What if the API calls to open AI fail or get rate limited. How is that surfaced. Will I see that in my observability. Will queries just silently miss results.

If the DB does the embedding itself synchronously within the write it would make sense. That would be more like elastic search or a typical full text index.

Re: Vector databases are the wrong abstraction

#40

Post by whakim »

This is actually really cool, and despite what I'm sure will come off as (constructive) criticism, I am very impressed!

First, I think you oversell the overhead of keeping data in sync and the costs of not doing so in a timely manner. Almost any distributed system that is using multiple databases already needs to have a strategy for dealing with inconsistent data. As far as this problem goes, inconsistent embeddings are a pretty minor issue given that (1) most embedding-based workflows don't do a lot of updating/deletion; and (2) the sheer volume of embeddings from only a small corpus of data means that in practice you're unlikely to notice consistency issues. In most cases you can get away with doing much less than is described in this post. That being said, I want to emphasize that I still think not having to worrying about syncing data is indeed cool.

Second, IME the most significant drawback to putting your embeddings in a Postgres database with all your other data is that the workload looks so different. To take one example, HNSW indices using pgvector consume a ton of resources - even a small index of tens of millions of embeddings may be hundreds of gigabytes on disk and requires very aggressive vacuuming to perform optimally. It's very easy to run into resource contention issues when you effectively have an index that will consume all the available system resources. The canonical solution is to move your data into another database, but then you've recreated the consistency problem that your solution purports to solve.

Third, a question: how does this interact with filtering? Can you take advantage of partial indices on the underlying data? Are some of the limitations in pgvector's HNSW implementation (as far as filtering goes) still present?

Post reply on HN