An early look at HNSW performance with pgvector
1–10 of 18 posts
Re: An early look at HNSW performance with pgvector
#2The results are very promising for the Postgres ecosystem, as the HNSW index shows a significant performance/recall improvement over the current ivfflat index in pgvector.
HNSW will be merged in v0.5.0. I can't speak for Andrew (the creator) but it seems that this release is imminent[0], pending some benchmarking and minor improvements. This is a first look at the performance of pgvector’s HNSW implementation at a specific commit[1].
[0] https://github.com/pgvector/pgvector/commit/51d292c93dff82f6...
Re: An early look at HNSW performance with pgvector
#3I like these measurements with ANN-benckmark! They allow to compare performance of different index implementations apples-to-apples i.e. at the same build parameters set rather than using some fixed settings (or, even worse, default settings that are different).
The blogpost is very thorough, lots of measurements of different datasets, including great 1536-dimensional 1M rows dbpedia-openai dataset. Furthermore, a very strong point is that all parameters and method is described and transparent.
Re: An early look at HNSW performance with pgvector
#4This may be a dumb question but with OpenAI embeddings do we need to use cosine similarity or is the simple distance equivalent? I used cosine similarity before but not sure.
Re: An early look at HNSW performance with pgvector
#5I like these measurements with ANN-benckmark! They allow to compare performance of different index implementations apples-to-apples i.e. at the same build parameters set rather than using some fixed settings (or, even worse, default settings that are different). The blogpost is very thorough, lots of measurements of different datasets, including great 1536-dimensional 1M rows dbpedia-openai dataset. Furthermore, a ve…
Blog author. Thanks for the analysis -- I agree that the ANN Benchmark does provide a nice framework for helping with apples-to-apples comparisons. In this case, being able to use the "--local" flag made it easier to run using the native environment, vs. putting it into a container. I'm looking for to ANN Benchmark having more datasets!
Re: An early look at HNSW performance with pgvector
#6This may be a dumb question but with OpenAI embeddings do we need to use cosine similarity or is the simple distance equivalent? I used cosine similarity before but not sure.
Blog author. You can choose to use any distance metrics. One reason cosine similarity is popular (and used) is that for many of these higher dimensional datasets, it gives a better representation of "nearness" across all the data basd on the nature of "angular" distance. But depending on how your data is distributed, something like L2 distance (Euclidean) could make more sense.
Re: An early look at HNSW performance with pgvector
#7This is pretty cool, but what are the scaling limits of a pgvector-based embeddings storage solution? 1 TB of embeddings? 100 TB? Is pgvector suitable for large scale installations?
Re: An early look at HNSW performance with pgvector
#8Can this HNSW implementation in PgVector be sharded across different nodes? Qdrant seems to make that easy.
Re: An early look at HNSW performance with pgvector
#9Glad to see more work on pgvector but why test on such small datasets on a large memory machine? The big ann datasets have 1B points and are much more interesting/representative of current embedding use cases (eg from dual encoder models).
I’m also curious if there is a way to not store everything in memory for pgvector. Is that possible?
Lastly, what is the parallelism story? Is it just using a thread pool under the hood? OpenMP?
Understanding if pgvector plans to support point insertions and deletions is also important in practice.
Re: An early look at HNSW performance with pgvector
#10Seems that pgvector has a viable competitor extension: https://github.com/tensorchord/pgvecto.rs