Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ANN v3: 200ms p99 query latency over 100B vectors

turbopuffer.com

Post reply on HN
Print view View original

21–30 of 49 posts

Re: ANN v3: 200ms p99 query latency over 100B vectors

#21

Post by lmeyerov »

kgeist wrote: post #6

Are there vector DBs with 100B vectors in production which work well? There was a paper which showed that there's 12% loss in accuracy at just 1 mln vectors. Maybe some kind of logical sharding is another option, to improve both accuracy and speed.

I don't know at these scales, but at the 1M-100M, we found switching from out-of-box embeddings to fine-tuning our embeddings gave less of a sting in the compression/recall trade-off . We had a 10-100X win here wrt comparable recall with better compression.

I'm not sure how that'd work with the binary quantization phase though. For example, we use Matroyska, and some of the bits matter way more than others, so that might be super painful.

Re: ANN v3: 200ms p99 query latency over 100B vectors

#24

Post by nvanbenschoten »

alanwli wrote: post #23

Out of curiosity, how is the 92% recall calculated? For a given query, is the recall compared to the true topk of all 100B vectors vs. recall at each of N shards compared to the topk of each respective shard?

(author here) The 92% mentioned in this post is showing recall@10 across all 100B vectors, calculated by comparing to the global top_k.

turbopuffer will also continuously monitor production recall at the per-shard level (or on-demand with https://turbopuffer.com/docs/recall). Perhaps counterintuitively, the global recall will actually be better than the per-shard recall if each shard is asked for its own, local top_k!

Re: ANN v3: 200ms p99 query latency over 100B vectors

#25

Post by enigmo »

benesch wrote: post #14
sa-code wrote: post #10

Earlier quoted context omitted.

Qdrant is also a good default choice, since it can work in-memory for development, with a hard drive for small deployments and also for "web scale" workloads. As a principal eng, side-stepping a migration and having a good local dev experience is too good of a deal to pass up. That being said, turbopuffer looks interesting. I will check it out. Hopefully their local dev experience is good

For local dev + testing, we recommend just hitting the production turbopuffer service directly, but with a separate test org/API key: https://turbopuffer.com/docs/testing Works well for the vast majority of our customers (although we get the very occasional complaint about wanting a dev environment that works offline). The dataset sizes for local dev are usually so small that the cost rounds to free.

having a local simulator (DynamoDB, Spanner, others) helps me a lot for offline/local development and CI. when a vendor doesn't off this I have often end up mocking it out (one way or another) and have to wait for integration or e2e tests for feedback that could have been pushed further to the left.

in many CI environments unit tests don't have network access, it's not purely a price consideration.

(not a turbopuffer customer but I have been looking at it)

Re: ANN v3: 200ms p99 query latency over 100B vectors

#26

Post by jauntywundrkind »

jascha_eng wrote: post #2

This is legitimately pretty impressive. I think the rule of thumb is now, go with postgres(pgvector) for vector search until it breaks, then go with turbopuffer.

I'd love to know how they compare versus MixedBread, what relative strengths each has. https://www.mixedbread.com/

I really really enjoy & learn a lot from the mixedbread blog. And they find good stuff to open source (although the product itself is closed). https://www.mixedbread.com/blog

I feel like there's a lot of overlap but also probably a lot of distinction too. Pretty new to this space of products though.

Re: ANN v3: 200ms p99 query latency over 100B vectors

#27

Post by benesch »

enigmo wrote: post #25
benesch wrote: post #14

Earlier quoted context omitted.

For local dev + testing, we recommend just hitting the production turbopuffer service directly, but with a separate test org/API key: https://turbopuffer.com/docs/testing Works well for the vast majority of our customers (although we get the very occasional complaint about wanting a dev environment that works offline). The dataset sizes for local dev are usually so small that the cost rounds to free.

having a local simulator (DynamoDB, Spanner, others) helps me a lot for offline/local development and CI. when a vendor doesn't off this I have often end up mocking it out (one way or another) and have to wait for integration or e2e tests for feedback that could have been pushed further to the left. in many CI environments unit tests don't have network access, it's not purely a price consideration. (not a turbopuffer…

> in many CI environments unit tests don't have network access, it's not purely a price consideration.

I've never seen a hard block on network access (how do you install packages/pull images?) but I am sympathetic to wanting to enforce that unit tests run quickly by minimizing/eliminating RTT to networked services.

We've considered the possibility of a local simulator before. Let me know if it winds up being a blocker for your use case.

Re: ANN v3: 200ms p99 query latency over 100B vectors

#28

Post by lambda »

benesch wrote: post #14
sa-code wrote: post #10

Earlier quoted context omitted.

Qdrant is also a good default choice, since it can work in-memory for development, with a hard drive for small deployments and also for "web scale" workloads. As a principal eng, side-stepping a migration and having a good local dev experience is too good of a deal to pass up. That being said, turbopuffer looks interesting. I will check it out. Hopefully their local dev experience is good

For local dev + testing, we recommend just hitting the production turbopuffer service directly, but with a separate test org/API key: https://turbopuffer.com/docs/testing Works well for the vast majority of our customers (although we get the very occasional complaint about wanting a dev environment that works offline). The dataset sizes for local dev are usually so small that the cost rounds to free.

> although we get the very occasional complaint about wanting a dev environment that works offline

It's only occasional because the people who care about dev environments that work offline are most likely to just skip you and move on.

For actual developer experience, as well as a number of use cases like customers with security and privacy concerns, being able to host locally is essential.

Fair enough if you don't care about those segments of the market, but don't confuse a small number of people asking about it with a small number of people wanting it.

Re: ANN v3: 200ms p99 query latency over 100B vectors

#29

Post by lambda »

benesch wrote: post #27
enigmo wrote: post #25

Earlier quoted context omitted.

having a local simulator (DynamoDB, Spanner, others) helps me a lot for offline/local development and CI. when a vendor doesn't off this I have often end up mocking it out (one way or another) and have to wait for integration or e2e tests for feedback that could have been pushed further to the left. in many CI environments unit tests don't have network access, it's not purely a price consideration. (not a turbopuffer…

> in many CI environments unit tests don't have network access, it's not purely a price consideration. I've never seen a hard block on network access (how do you install packages/pull images?) but I am sympathetic to wanting to enforce that unit tests run quickly by minimizing/eliminating RTT to networked services. We've considered the possibility of a local simulator before. Let me know if it winds up being a blocke…

> how do you install packages/pull images

You pre-build the images with packages installed beforehand, then use those image offline.

Re: ANN v3: 200ms p99 query latency over 100B vectors

#30

Post by benesch »

lambda wrote: post #29
benesch wrote: post #27

Earlier quoted context omitted.

> in many CI environments unit tests don't have network access, it's not purely a price consideration. I've never seen a hard block on network access (how do you install packages/pull images?) but I am sympathetic to wanting to enforce that unit tests run quickly by minimizing/eliminating RTT to networked services. We've considered the possibility of a local simulator before. Let me know if it winds up being a blocke…

> how do you install packages/pull images You pre-build the images with packages installed beforehand, then use those image offline.

My point is it's enough of a hassle to set up that I've yet to see that level of restriction in practice (across hundreds of CI systems).
Post reply on HN