Earlier quoted context omitted.
the solution described in the blog post is currently in production at 100B vectors
For what/who?
ANN v3: 200ms p99 query latency over 100B vectors
11–20 of 49 posts
Re: ANN v3: 200ms p99 query latency over 100B vectors
#12Re: ANN v3: 200ms p99 query latency over 100B vectors
#13Re: ANN v3: 200ms p99 query latency over 100B vectors
#14This is legitimately pretty impressive. I think the rule of thumb is now, go with postgres(pgvector) for vector search until it breaks, then go with turbopuffer.
Qdrant is also a good default choice, since it can work in-memory for development, with a hard drive for small deployments and also for "web scale" workloads. As a principal eng, side-stepping a migration and having a good local dev experience is too good of a deal to pass up. That being said, turbopuffer looks interesting. I will check it out. Hopefully their local dev experience is good
For local dev + testing, we recommend just hitting the production turbopuffer service directly, but with a separate test org/API key: https://turbopuffer.com/docs/testing
Works well for the vast majority of our customers (although we get the very occasional complaint about wanting a dev environment that works offline). The dataset sizes for local dev are usually so small that the cost rounds to free.
Re: ANN v3: 200ms p99 query latency over 100B vectors
#15v cool and impressive!
[deleted]
Re: ANN v3: 200ms p99 query latency over 100B vectors
#16Earlier quoted context omitted.
Qdrant is also a good default choice, since it can work in-memory for development, with a hard drive for small deployments and also for "web scale" workloads. As a principal eng, side-stepping a migration and having a good local dev experience is too good of a deal to pass up. That being said, turbopuffer looks interesting. I will check it out. Hopefully their local dev experience is good
For local dev + testing, we recommend just hitting the production turbopuffer service directly, but with a separate test org/API key: https://turbopuffer.com/docs/testing Works well for the vast majority of our customers (although we get the very occasional complaint about wanting a dev environment that works offline). The dataset sizes for local dev are usually so small that the cost rounds to free.
That’s not local though
Re: ANN v3: 200ms p99 query latency over 100B vectors
#17For those of us who operate on site, we have to add back network latency, which negates this win entirely and makes a proprietary cloud solution like this a nonstarter.
Often not a dealbreaker, actually! We can spin up new tpuf regions and procure dedicated interconnects to minimize latency to the on-prem network on request (and we have done this).
When you're operating at the 100B scale, you're pushing beyond the capacity that most on-prem setups can handle. Most orgs have no choice but to put a 100B workload into the nearest public cloud. (For smaller workloads, considerations are different, for sure.)
Re: ANN v3: 200ms p99 query latency over 100B vectors
#18Using Hierarchical Clustering significantly reduces recall; this is a solution we used and abandoned three years ago.
Re: ANN v3: 200ms p99 query latency over 100B vectors
#19This is legitimately pretty impressive. I think the rule of thumb is now, go with postgres(pgvector) for vector search until it breaks, then go with turbopuffer.
Qdrant is also a good default choice, since it can work in-memory for development, with a hard drive for small deployments and also for "web scale" workloads. As a principal eng, side-stepping a migration and having a good local dev experience is too good of a deal to pass up. That being said, turbopuffer looks interesting. I will check it out. Hopefully their local dev experience is good
Qdrant is one of the few vendors I actively steer people away from. Look at the GitHub issues, look at what their CEO says, look at their fake “advancements” that they pay for publicity on…
The number of people I know who’ve had unrecoverable shard failures on Qdrant is too high to take it seriously.
Re: ANN v3: 200ms p99 query latency over 100B vectors
#20Are there vector DBs with 100B vectors in production which work well? There was a paper which showed that there's 12% loss in accuracy at just 1 mln vectors. Maybe some kind of logical sharding is another option, to improve both accuracy and speed.
So many missing details...
Different vector indexes have very different recall and even different parameters for each dramatically impact this.
HNSW can have very good recall even at high vector counts.
There's also the embedding model, whether you're quantizing, if it's pure rag vs hybrid bm25 / static word embeddings vs graph connections, whether you're reranking etc etc