Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An early look at HNSW performance with pgvector

jkatz05.com

Post reply on HN
Print view View original

11–18 of 18 posts

Re: An early look at HNSW performance with pgvector

#11

Post by pashkinelfe »

moab wrote: post #9

Glad to see more work on pgvector but why test on such small datasets on a large memory machine? The big ann datasets have 1B points and are much more interesting/representative of current embedding use cases (eg from dual encoder models). I’m also curious if there is a way to not store everything in memory for pgvector. Is that possible? Lastly, what is the parallelism story? Is it just using a thread pool under the…

Pgvector doesn't need to store everything in memory. It behaves similar to almost any Postgres AM's and store index data on disk. Performance-wise it's better to have enough memory for index data remain in shared memory buffers, but it is not a requirement for pgvector.

Re: An early look at HNSW performance with pgvector

#12

Post by pashkinelfe »

moab wrote: post #9

Glad to see more work on pgvector but why test on such small datasets on a large memory machine? The big ann datasets have 1B points and are much more interesting/representative of current embedding use cases (eg from dual encoder models). I’m also curious if there is a way to not store everything in memory for pgvector. Is that possible? Lastly, what is the parallelism story? Is it just using a thread pool under the…

If you have 10M or bigger dataset of real-world OpenAI- dimensional vectors, please share, I'll use it in the next benchmarks. Random datasets are too misleading for vector search benchmarks because all ANN engines make use of internal distributions in datasets to struggle with the curse of dimensionality. So I never use random datasets for ann indexed benchmarking. Using simplified less dimensional (eg 128 instead of 1536) vectors also changes performance trends.

Re: An early look at HNSW performance with pgvector

#13

Post by moab »

pashkinelfe wrote: post #12
moab wrote: post #9

Glad to see more work on pgvector but why test on such small datasets on a large memory machine? The big ann datasets have 1B points and are much more interesting/representative of current embedding use cases (eg from dual encoder models). I’m also curious if there is a way to not store everything in memory for pgvector. Is that possible? Lastly, what is the parallelism story? Is it just using a thread pool under the…

If you have 10M or bigger dataset of real-world OpenAI- dimensional vectors, please share, I'll use it in the next benchmarks. Random datasets are too misleading for vector search benchmarks because all ANN engines make use of internal distributions in datasets to struggle with the curse of dimensionality. So I never use random datasets for ann indexed benchmarking. Using simplified less dimensional (eg 128 instead o…

See https://big-ann-benchmarks.com/neurips21.html

They're not OpenAI embeddings, but they are realistic, and much larger (number of vectors).

I think many production embeddings at non-OpenAI companies will use lower-dimensional vectors than 1536, so it makes sense to focus on non-OpenAI embeddings as well in your benchmarking.

Re: An early look at HNSW performance with pgvector

#14

Post by moab »

pashkinelfe wrote: post #11
moab wrote: post #9

Glad to see more work on pgvector but why test on such small datasets on a large memory machine? The big ann datasets have 1B points and are much more interesting/representative of current embedding use cases (eg from dual encoder models). I’m also curious if there is a way to not store everything in memory for pgvector. Is that possible? Lastly, what is the parallelism story? Is it just using a thread pool under the…

Pgvector doesn't need to store everything in memory. It behaves similar to almost any Postgres AM's and store index data on disk. Performance-wise it's better to have enough memory for index data remain in shared memory buffers, but it is not a requirement for pgvector.

Thanks for the response. I wonder whether HNSW will still perform well if it needs to page neighbor-lists to/from disk. Do you plan to benchmark the setting where the dataset is too large to fit in-memory?

Re: An early look at HNSW performance with pgvector

#16

Post by kiwicopple »

nicwolff wrote: post #15
Dowwie wrote: post #10

Seems that pgvector has a viable competitor extension: https://github.com/tensorchord/pgvecto.rs

A couple: https://github.com/neondatabase/pg_embedding

the charts in this blog post show benchmarks with pgvectors HNSW vs pg_embedding.

Re: An early look at HNSW performance with pgvector

#17

Post by kiwicopple »

heyitsguay wrote: post #7

This is pretty cool, but what are the scaling limits of a pgvector-based embeddings storage solution? 1 TB of embeddings? 100 TB? Is pgvector suitable for large scale installations?

The scalability characteristics for HNSW haven’t been properly benchmarked yet. If it’s anything like ivfflat then it should be reasonably predictable based on memory size: https://supabase.com/blog/pgvector-performance

I think 100TB is getting more into “sharded postgres” territory

Re: An early look at HNSW performance with pgvector

#18

Post by jkatz05 »

heyitsguay wrote: post #7

This is pretty cool, but what are the scaling limits of a pgvector-based embeddings storage solution? 1 TB of embeddings? 100 TB? Is pgvector suitable for large scale installations?

Blog author. I've done some separate testing on storing ~500GB of embeddings (~1B embeddings) in a partitioned table. The partition key was built using IVFFLAT as a "coarse quantizer" (in this case, sampling the entire dataset and finding K means), storing the mean vectors in a separate table, and then loading each vector into the partition with closest center. After that, I built an IVFFLAT index on each partition. With the indexes, this added up to ~1TB storage. This was primarily a "is it possible test" vs. thorough benchmarking.
Post reply on HN