Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Case Against PGVector

alex-jacobs.com

Post reply on HN
Print view View original

91–100 of 144 posts

Re: The Case Against PGVector

#91

Post by simonw »

morshu9001 wrote: post #81
simonw wrote: post #74

Earlier quoted context omitted.

And honestly, even then YAGNI might still win. There's a big opportunity cost involved in optimizing prematurely. 9/10 times you're wasting your time, and you may have found product-market fit faster if you had spent that time trying out other feature ideas instead. If you hit a point where you have to do a painful migration because your product is succeeding that's a point to be celebrated in my opinion. You might n…

Yeah the "only if" is more like a "necessary, not sufficient." The future migration pain had better be extremely bad to worry about it so far in advance. Or it should be a well defined problem. It's easier to determine the right solution after you've already encountered the problem, maybe in a past project. If you're unsure, just keep your options open.

A few years ago I coined the term PAGNI for "Probably Are Gonna Need It" to cover things that are worth putting in there from the start because they're relatively cheap to implement early but quite expensive to add later on: https://simonwillison.net/2021/Jul/1/pagnis/

Re: The Case Against PGVector

#92

Post by hunterpayne »

tacoooooooo wrote: post #25
rudderdev wrote: post #12

As others have commented, all the mentioned issues are resolved, I will favour in using the PGVector. If Postgres can be a good choice over Kafka to deliver 100k events/sec [1], then why not PGVector over Chroma or other specialized vector search (unless there is a specific requirement that can't be solved wit minor code/config changes)! [1] Ref: https://news.ycombinator.com/item?id=44659678

how are all of the mentioned issues resolved?

So its a longish article and doing a point by point explanation is probably too much for a single post. But several of the points are solved but just standing up a specific Postgres instance for the vector use cases instead of doing this inside an existing instance.

Most of the rest of his complaints comes down to this is complex stuff. True, but its not a solution, its a tool used in making a solution. So when using pg_vector directly, you probably need to understand databases to a more significant degree than a custom solution that won't work for you the moment your requirements change. You surely need to understand databases more than the author does. He doesn't point to a single thing that pg_vector doesn't do or doesn't do well. He just complains it hard to do.

In summary, pg_vector is a toolkit for building vector based functionality, not a custom solution for a specific use case. What is best for you comes down to your team's skills and expertise with databases and if your specific requirements will change. Choose poorly and it could go very badly.

Re: The Case Against PGVector

#93

Post by bashtoni »

pqdbr wrote: post #32

Id love to read a blog post like this about S3 Vector buckets. Does anyone have experience with it in production?

The service is still in preview, so AWS are explicitly telling people not to put it into production.

From my non-production experiments with it, the main limitation is that you can only retrieve up to 30 top_k results, which means you can't use it with a re-ranker, or at least not as effectively. For many production use cases that will be a deal breaker.

Re: The Case Against PGVector

#95

Post by sa-code »

simonw wrote: post #88
FuckButtons wrote: post #82

Earlier quoted context omitted.

why is this amazing, it’s just a 1 bit lossy compression representation of the original information? If you have a vector in n-dimensional space this is effectively just representing the basis vectors that the original has.

You can take 8192 bytes of information (1024 x 32 bit floats) and reduce that to 128 bytes (1024 bits, a 64x reduction in size!) and still get results that are about 95% as good. I find that cool and surprising.

I'm with you, it's very satisfying to see a simple technique work well. It's impressive

Re: The Case Against PGVector

#96

Post by kevstev »

Fripplebubby wrote: post #75
simonw wrote: post #74

Earlier quoted context omitted.

And honestly, even then YAGNI might still win. There's a big opportunity cost involved in optimizing prematurely. 9/10 times you're wasting your time, and you may have found product-market fit faster if you had spent that time trying out other feature ideas instead. If you hit a point where you have to do a painful migration because your product is succeeding that's a point to be celebrated in my opinion. You might n…

I think I see this point now. I thought of YAGNI as, "don't ever over-engineer because you get it wrong a lot of the time" but really, "don't over-engineer out of the gate and be thankful if you get a chance to come back and do it right later". That fits my case exactly, and that's what we did (and it wasn't actually that painful to migrate).

At my last job I took over eng at a Series B startup, and my (non-technical) CEO was an ill tempered type and pretty much wanted me to tell him that the entire tech stack was shit and the previous architect/pseudo head of eng was shit, etc. And I was like no... some tradeoffs were made that make a ton of sense for an early stage startup, and the great news is that you are still here and now have the revenue and customer base to start thinking in terms of building things for the next 3-5 years, even though some of things are starting to break. And even better, nothing was so dire that it required stopping the world, we could continue to build and shore up some of the struggling things at the same time.

He seemed to really want me to blame everything on my predecessor and call some kind of crisis, and seemed annoyed by my analysis, which was confusing at the time. But yeah, there are absolutely tradeoffs you make early in a startups life, you just have to know where to take shortcuts and where you at least leave the architecture open to scaling. My biggest critique is that they were at least a year, if not two, past the point where they should have left ultra scrappy startup mode that just throws things at the wall and started building with a longer view.

I have also seen a friend build out a flawless architecture ready to scale to millions of users, but never got close to a product fit. I felt he wasted at least 6 months building out all this infra scaffolding for nothing.

Re: The Case Against PGVector

#97

Post by whakim »

xfalcox wrote: post #3

> Nobody’s actually run this in production We do at Discourse, in thousands of databases, and it's leveraged in most of the billions of page views we serve. > Pre- vs. Post-Filtering (or: why you need to become a query planner expert) This was fixed in version 0.8.0 via Iterative Scans ( https://github.com/pgvector/pgvector?tab=readme-ov-file#iter... ) > Just use a real vector database If you are running a single ser…

Interested to hear more about your experience here. At Halcyon, we have trillions of embeddings and found Postgres to be unsuitable at several orders of magnitude less than we currently have.

On the iterative scan side, how do you prevent this from becoming too computationally intensive with a restrictive pre-filter, or simply not working at all? We use Vespa, which means effectively doing a map-reduce across all of our nodes; the effective number of graph traversals to do is smaller, and the computational burden mostly involves scanning posting lists on a per-node basis. I imagine to do something similar in postgres, you'd need sharded tables, and complicated application logic to control what you're actually searching.

How do you deal with re-indexing and/or denormalizing metadata for filtering? Do you simply accept that it'll take hours or days?

I agree with you, however, that vector databases are not a panacea (although they do remove a huge amount of devops work, which is worth a lot!). Vespa supports filtering across parent-child relationships (like a relational database) which means we don't have to reindex a trillion things every time we want to add a new type of filter, which with a previous vector database vendor we used took us almost a week.

Re: The Case Against PGVector

#98

Post by xfalcox »

simonw wrote: post #72
xfalcox wrote: post #18

Earlier quoted context omitted.

Also worth mentioning that we use quantization extensively: - halfvec (16bit float) for storage - bit (binary vectors) for indexes Which makes the storage cost and on-going performance good enough that we could enable this in all our hosting.

It still amazes me that the binary trick works. For anyone who hasn't seen it yet: it turns out many embedding vectors of e.g. 1024 floating point numbers can be reduced to a single bit per value that records if it's higher or lower than 0... and in this reduced form much of the embedding math still works! This means you can e.g. filter to the top 100 using extremely memory efficient and fast bit vectors, then run a…

I was taken back when I saw what was basically zero recall loss in the real world task of finding related topics, by doing the same thing you described where we over capture with binary embeddings, and only use the full (or half) precision on the subset.

Making the storage cost of the index 32 times smaller is the difference of being able to offer this at scale without worrying too much about the overhead.

Re: The Case Against PGVector

#99

Post by whakim »

sgarland wrote: post #14

> The problem is that index builds are memory-intensive operations, and Postgres doesn’t have a great way to throttle them. maintenance_work_mem begs to differ. > You rebuild the index periodically to fix this, but during the rebuild (which can take hours for large datasets), what do you do with new inserts? Queue them? Write to a separate unindexed table and merge later? You use REINDEX CONCURRENTLY. > But updating…

> maintenance_work_mem begs to differ.

HNSW indices are big. Let's suppose I have an HNSW index which fits in a few hundred gigabytes of memory, or perhaps a few terabytes. How do I reasonably rebuild this using maintenance_work_mem? Double the size of my database for a week? What about the knock-on impacts on the performance for the rest of my database-stuff - presumably I'm relying on this memory for shared_buffers and caching? This seems like the type of workload that is being discussed here, not a toy 20GB index or something.

> You use REINDEX CONCURRENTLY.

Even with a bunch of worker processes, how do I do this within a reasonable timeframe?

> How do you think a B+tree gets updated?

Sure, the computational complexity of insertion into an HNSW index is sublinear, the constant factors are significant and do actually add up. That being said, I do find this the weakest of the author's arguments.

Re: The Case Against PGVector

#100

Post by xfalcox »

whakim wrote: post #97
xfalcox wrote: post #3

> Nobody’s actually run this in production We do at Discourse, in thousands of databases, and it's leveraged in most of the billions of page views we serve. > Pre- vs. Post-Filtering (or: why you need to become a query planner expert) This was fixed in version 0.8.0 via Iterative Scans ( https://github.com/pgvector/pgvector?tab=readme-ov-file#iter... ) > Just use a real vector database If you are running a single ser…

Interested to hear more about your experience here. At Halcyon, we have trillions of embeddings and found Postgres to be unsuitable at several orders of magnitude less than we currently have. On the iterative scan side, how do you prevent this from becoming too computationally intensive with a restrictive pre-filter, or simply not working at all? We use Vespa, which means effectively doing a map-reduce across all of…

We host thousands of forums but each one has its own database, which means we get a sort of free sharding of the data where each instance has less than a million topics on average.

I can totally see that at a trillion scale for a single shard you want a specialized dedicated service, but that is also true for most things in tech when you get to the extreme scale .

Post reply on HN