Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Case Against PGVector

alex-jacobs.com

Post reply on HN
Print view View original

81–90 of 144 posts

Re: The Case Against PGVector

#81

Post by morshu9001 »

simonw wrote: post #74
morshu9001 wrote: post #71

Earlier quoted context omitted.

If the consequence of being wrong about the scalability is that you just have to migrate later instead of sooner, that's a win for YAGNI. It's only a loss if hitting this limit later causes service disruption or makes the migration way harder than if you'd done it sooner.

And honestly, even then YAGNI might still win. There's a big opportunity cost involved in optimizing prematurely. 9/10 times you're wasting your time, and you may have found product-market fit faster if you had spent that time trying out other feature ideas instead. If you hit a point where you have to do a painful migration because your product is succeeding that's a point to be celebrated in my opinion. You might n…

Yeah the "only if" is more like a "necessary, not sufficient." The future migration pain had better be extremely bad to worry about it so far in advance.

Or it should be a well defined problem. It's easier to determine the right solution after you've already encountered the problem, maybe in a past project. If you're unsure, just keep your options open.

Re: The Case Against PGVector

#82

Post by FuckButtons »

simonw wrote: post #72
xfalcox wrote: post #18

Earlier quoted context omitted.

Also worth mentioning that we use quantization extensively: - halfvec (16bit float) for storage - bit (binary vectors) for indexes Which makes the storage cost and on-going performance good enough that we could enable this in all our hosting.

It still amazes me that the binary trick works. For anyone who hasn't seen it yet: it turns out many embedding vectors of e.g. 1024 floating point numbers can be reduced to a single bit per value that records if it's higher or lower than 0... and in this reduced form much of the embedding math still works! This means you can e.g. filter to the top 100 using extremely memory efficient and fast bit vectors, then run a…

why is this amazing, it’s just a 1 bit lossy compression representation of the original information? If you have a vector in n-dimensional space this is effectively just representing the basis vectors that the original has.

Re: The Case Against PGVector

#83

Post by jascha_eng »

xfalcox wrote: post #3

> Nobody’s actually run this in production We do at Discourse, in thousands of databases, and it's leveraged in most of the billions of page views we serve. > Pre- vs. Post-Filtering (or: why you need to become a query planner expert) This was fixed in version 0.8.0 via Iterative Scans ( https://github.com/pgvector/pgvector?tab=readme-ov-file#iter... ) > Just use a real vector database If you are running a single ser…

There are also approaches do doing the filtering while traversing a vector index (not just pre/post) e.g. this paper by microsoft explains an approach https://dl.acm.org/doi/10.1145/3543507.3583552 which pgvectorscale implements here: https://github.com/timescale/pgvectorscale?tab=readme-ov-fil...

In theory these can be more efficient than plain pre/post filtering.

Re: The Case Against PGVector

#84

Post by dpflan »

xfalcox wrote: post #51
dpflan wrote: post #33

Earlier quoted context omitted.

What are you using it for? Is it part of a hybrid search system (keyword + vector)?

In Discourse embeddings power: - Related Topics, a list of topics to read next, which uses embeddings of the current topic as the key to search for similar ones - Suggesting tags and categories when composing a new topic - Augmented search - RAG for uploaded files

Thanks for the details. Also, always appreciated Discord's engineering blog posts. Lots of interesting stories, and nice to see a company discuss using Elixir at scale.

Re: The Case Against PGVector

#85

Post by semiquaver » 

  > You rebuild the index periodically to fix this, but during the rebuild (which can take hours for large datasets), what do you do with new inserts? Queue them? Write to a separate unindexed table and merge later?
What is wrong with REINDEX CONCURRENTLY?

Re: The Case Against PGVector

#86

Post by jankovicsandras »

Shameless plug: https://github.com/jankovicsandras/plpgsql_bm25 BM25 search implemented in PL/pgSQL ( Unlicense / Public domain )

The repo includes plpgsql_bm25rrf.sql : PL/pgSQL function for Hybrid search ( plpgsql_bm25 + pgvector ) with Reciprocal Rank Fusion; and Jupyter notebook examples.

Re: The Case Against PGVector

#87

Post by kgeist »

bob1029 wrote: post #60

I'm still stuck on whether or not vector search (regardless of vendor) is actually the right way to solve the kinds of problems that everyone seems to believe it's great at. BM25 with query rewriting & expansion can do a lot of heavy lifting if you invest any time at all in configuring things to match your problem space. The article touches on FTS engines and hybrid approaches, but I would start there. Figure out whe…

I'm currently building RAG for our product (using Lucene). What I've found is that embeddings alone don't help much. With hybrid search (BM25+HNSW) they gave me only like +10% boost compared to BM25 alone (on average). In my evaluation datasets, the only case where they helped tremendously was for cases like "a user asks a question in French but the documents are all in English", it went from 6% retrieval to 65% on some datasets.

I got a significant boost (from 65% on average to over 80%) by adding a proper reranker and query rewriting (3 additional phrases to search for).

I think embeddings are overrated in that blog posts often make you believe they are the end of the story. What I've found is that they should be rather treated as a lightweight filtering/screening tool to quickly find a pool of candidates as a first stage, before you do the actual stuff (apply a reranker). If BM25 already works as well as a pre-filtering tool, you don't even need embeddings (with all the indexing headaches).

Re: The Case Against PGVector

#88

Post by simonw »

FuckButtons wrote: post #82
simonw wrote: post #72

Earlier quoted context omitted.

It still amazes me that the binary trick works. For anyone who hasn't seen it yet: it turns out many embedding vectors of e.g. 1024 floating point numbers can be reduced to a single bit per value that records if it's higher or lower than 0... and in this reduced form much of the embedding math still works! This means you can e.g. filter to the top 100 using extremely memory efficient and fast bit vectors, then run a…

why is this amazing, it’s just a 1 bit lossy compression representation of the original information? If you have a vector in n-dimensional space this is effectively just representing the basis vectors that the original has.

You can take 8192 bytes of information (1024 x 32 bit floats) and reduce that to 128 bytes (1024 bits, a 64x reduction in size!) and still get results that are about 95% as good.

I find that cool and surprising.

Re: The Case Against PGVector

#89

Post by tacoooooooo »

jascha_eng wrote: post #83
xfalcox wrote: post #3

> Nobody’s actually run this in production We do at Discourse, in thousands of databases, and it's leveraged in most of the billions of page views we serve. > Pre- vs. Post-Filtering (or: why you need to become a query planner expert) This was fixed in version 0.8.0 via Iterative Scans ( https://github.com/pgvector/pgvector?tab=readme-ov-file#iter... ) > Just use a real vector database If you are running a single ser…

There are also approaches do doing the filtering while traversing a vector index (not just pre/post) e.g. this paper by microsoft explains an approach https://dl.acm.org/doi/10.1145/3543507.3583552 which pgvectorscale implements here: https://github.com/timescale/pgvectorscale?tab=readme-ov-fil... In theory these can be more efficient than plain pre/post filtering.

pgvectorscale is not available in RDS so this wasnt a great solution for us! but it does likely solve many of the problems with vanilla pgvector (what this post was about)

Re: The Case Against PGVector

#90

Post by auraham »

jjfoooo4 wrote: post #29

When using vectors / embeddings models, I think there's a lot of low hanging fruit to be had with non-massive datasets - your support documentation, your product info, a lot of search use cases. For these, the interface I really want is more like a file system than a database - I want to be able to just write and update documents like a file system and have the indexes update automatically and invisibly. So basically…

Have you tried cocoindex?

[1] https://cocoindex.io/

[2] https://dev.to/cocoindex/how-to-build-index-with-text-embedd...

Post reply on HN