Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ANN v3: 200ms p99 query latency over 100B vectors

turbopuffer.com

Post reply on HN
Print view View original

31–40 of 49 posts

Re: ANN v3: 200ms p99 query latency over 100B vectors

#31

Post by sa-code »

benesch wrote: post #14
sa-code wrote: post #10

Earlier quoted context omitted.

Qdrant is also a good default choice, since it can work in-memory for development, with a hard drive for small deployments and also for "web scale" workloads. As a principal eng, side-stepping a migration and having a good local dev experience is too good of a deal to pass up. That being said, turbopuffer looks interesting. I will check it out. Hopefully their local dev experience is good

For local dev + testing, we recommend just hitting the production turbopuffer service directly, but with a separate test org/API key: https://turbopuffer.com/docs/testing Works well for the vast majority of our customers (although we get the very occasional complaint about wanting a dev environment that works offline). The dataset sizes for local dev are usually so small that the cost rounds to free.

I should have clarified, by local dev and testing I did in fact mean offline usage.

Without that it’s unfortunately a non starter

Re: ANN v3: 200ms p99 query latency over 100B vectors

#32

Post by sa-code »

nostrebored wrote: post #19
sa-code wrote: post #10

Earlier quoted context omitted.

Qdrant is also a good default choice, since it can work in-memory for development, with a hard drive for small deployments and also for "web scale" workloads. As a principal eng, side-stepping a migration and having a good local dev experience is too good of a deal to pass up. That being said, turbopuffer looks interesting. I will check it out. Hopefully their local dev experience is good

Qdrant is one of the few vendors I actively steer people away from. Look at the GitHub issues, look at what their CEO says, look at their fake “advancements” that they pay for publicity on… The number of people I know who’ve had unrecoverable shard failures on Qdrant is too high to take it seriously.

I’m curious about this. Could you please point to some things the CEO has said, or reports of shard failures?

The bit about paying for publicity doesn’t bother me.

Edit: I haven’t found anything egregious that the CEO has said, or anything really sketchy. The shard failure warnings look serious, but the issues look closed

https://github.com/qdrant/qdrant/issues/6025

https://github.com/qdrant/qdrant/issues/4939

Re: ANN v3: 200ms p99 query latency over 100B vectors

#34

Post by benesch »

sa-code wrote: post #31
benesch wrote: post #14

Earlier quoted context omitted.

For local dev + testing, we recommend just hitting the production turbopuffer service directly, but with a separate test org/API key: https://turbopuffer.com/docs/testing Works well for the vast majority of our customers (although we get the very occasional complaint about wanting a dev environment that works offline). The dataset sizes for local dev are usually so small that the cost rounds to free.

I should have clarified, by local dev and testing I did in fact mean offline usage. Without that it’s unfortunately a non starter

So I can note this down on our roadmap, what's the root of your requirement here? Supporting local dev without internet (airplanes, coffee shops, etc.)? Unit test speed? Something else?

Re: ANN v3: 200ms p99 query latency over 100B vectors

#35

Post by sa-code »

lambda wrote: post #28
benesch wrote: post #14

Earlier quoted context omitted.

For local dev + testing, we recommend just hitting the production turbopuffer service directly, but with a separate test org/API key: https://turbopuffer.com/docs/testing Works well for the vast majority of our customers (although we get the very occasional complaint about wanting a dev environment that works offline). The dataset sizes for local dev are usually so small that the cost rounds to free.

> although we get the very occasional complaint about wanting a dev environment that works offline It's only occasional because the people who care about dev environments that work offline are most likely to just skip you and move on. For actual developer experience, as well as a number of use cases like customers with security and privacy concerns, being able to host locally is essential. Fair enough if you don't ca…

Can confirm. With a setup that works offline, one can

- start small on a laptop. Going through procurement at companies is a pain

- test things in CI reliably. Outages don’t break builds

- transition from laptop scale to web scale easily with the same API with just a different backend

Otherwise it’s really hard to justify not using S3 vectors here

The current dev experience is to start with faiss for PoCs, move to pgvector and then something heavy duty like one of the Lucene wrappers.

Re: ANN v3: 200ms p99 query latency over 100B vectors

#36

Post by sa-code »

benesch wrote: post #34
sa-code wrote: post #31

Earlier quoted context omitted.

I should have clarified, by local dev and testing I did in fact mean offline usage. Without that it’s unfortunately a non starter

So I can note this down on our roadmap, what's the root of your requirement here? Supporting local dev without internet (airplanes, coffee shops, etc.)? Unit test speed? Something else?

I listed some reasons in another comment: https://news.ycombinator.com/item?id=46757853

I appreciate your responsiveness and open mind

Re: ANN v3: 200ms p99 query latency over 100B vectors

#37

Post by nik9000 »

lambda wrote: post #28
benesch wrote: post #14

Earlier quoted context omitted.

For local dev + testing, we recommend just hitting the production turbopuffer service directly, but with a separate test org/API key: https://turbopuffer.com/docs/testing Works well for the vast majority of our customers (although we get the very occasional complaint about wanting a dev environment that works offline). The dataset sizes for local dev are usually so small that the cost rounds to free.

> although we get the very occasional complaint about wanting a dev environment that works offline It's only occasional because the people who care about dev environments that work offline are most likely to just skip you and move on. For actual developer experience, as well as a number of use cases like customers with security and privacy concerns, being able to host locally is essential. Fair enough if you don't ca…

As someone who works for a competitor, they are probably right holding off on that segment for a while. Supporting both cloud and local deployments is somewhere between 20% harder and 300% harder depending on the day.

I'm watching them with excitement. We all learn from each other. There's so much to do.

Re: ANN v3: 200ms p99 query latency over 100B vectors

#39

Post by dzbarsky »

benesch wrote: post #30
lambda wrote: post #29

Earlier quoted context omitted.

> how do you install packages/pull images You pre-build the images with packages installed beforehand, then use those image offline.

My point is it's enough of a hassle to set up that I've yet to see that level of restriction in practice (across hundreds of CI systems).

Look into Bazel, a very standard build system used at many large tech companies. It splits fetches from build/test actions and allows blocking network for build/test actions with a single CLI flag. No hassle at all.

The fact that you haven't come across this kind of setup suggests that your hundreds of CI systems are not representative of the industry as a whole.

Re: ANN v3: 200ms p99 query latency over 100B vectors

#40

Post by hwspeed »

The offline/local dev point is underrated. Being able to iterate without network latency or metered API costs makes a huge difference for prototyping. The challenge is making sure your local setup actually matches prod behavior. I've been burned by pgvector working fine locally then hitting performance cliffs at scale when the index doesn't fit in memory anymore.
Post reply on HN