ANN v3: 200ms p99 query latency over 100B vectors
turbopuffer.com
ANN v3: 200ms p99 query latency over 100B vectors
1–10 of 49 posts
Re: ANN v3: 200ms p99 query latency over 100B vectors
#2This is legitimately pretty impressive. I think the rule of thumb is now, go with postgres(pgvector) for vector search until it breaks, then go with turbopuffer.
Re: ANN v3: 200ms p99 query latency over 100B vectors
#3This is legitimately pretty impressive. I think the rule of thumb is now, go with postgres(pgvector) for vector search until it breaks, then go with turbopuffer.
seems like a good rule of thumb to me! though i would perhaps lump "cost" into the "until it breaks" equation. even with decent perf, pg_vector's economics can be much worse, especially in multi-tenant scenarios where you need many small indexes (this is true of any vector db that builds indexes primarily on RAM/SSD)
Re: ANN v3: 200ms p99 query latency over 100B vectors
#4v cool and impressive!
Re: ANN v3: 200ms p99 query latency over 100B vectors
#5Fun!
I was curious given the cloud discussion - a quick search suggests default AWS SSD bandwidth is 250 MB/s, and you can pay more for 1 GB/s. Similar for s3, one http connection is < 100 MB/s, and you can pay for more parallel connections. So the hot binary quantized search index is doing a lot of work to minimize these both for the initial hot queries and pruning later fetches. Very cool!
Re: ANN v3: 200ms p99 query latency over 100B vectors
#6Are there vector DBs with 100B vectors in production which work well? There was a paper which showed that there's 12% loss in accuracy at just 1 mln vectors. Maybe some kind of logical sharding is another option, to improve both accuracy and speed.
Re: ANN v3: 200ms p99 query latency over 100B vectors
#7Are there vector DBs with 100B vectors in production which work well? There was a paper which showed that there's 12% loss in accuracy at just 1 mln vectors. Maybe some kind of logical sharding is another option, to improve both accuracy and speed.
the solution described in the blog post is currently in production at 100B vectors
Re: ANN v3: 200ms p99 query latency over 100B vectors
#8For those of us who operate on site, we have to add back network latency, which negates this win entirely and makes a proprietary cloud solution like this a nonstarter.
Re: ANN v3: 200ms p99 query latency over 100B vectors
#9Are there vector DBs with 100B vectors in production which work well? There was a paper which showed that there's 12% loss in accuracy at just 1 mln vectors. Maybe some kind of logical sharding is another option, to improve both accuracy and speed.
the solution described in the blog post is currently in production at 100B vectors
For what/who?
Re: ANN v3: 200ms p99 query latency over 100B vectors
#10This is legitimately pretty impressive. I think the rule of thumb is now, go with postgres(pgvector) for vector search until it breaks, then go with turbopuffer.
Qdrant is also a good default choice, since it can work in-memory for development, with a hard drive for small deployments and also for "web scale" workloads.
As a principal eng, side-stepping a migration and having a good local dev experience is too good of a deal to pass up.
That being said, turbopuffer looks interesting. I will check it out. Hopefully their local dev experience is good