Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ANN v3: 200ms p99 query latency over 100B vectors

turbopuffer.com

Post reply on HN
Print view View original

41–49 of 49 posts

Re: ANN v3: 200ms p99 query latency over 100B vectors

#41

Post by benesch »

lambda wrote: post #28
benesch wrote: post #14

Earlier quoted context omitted.

For local dev + testing, we recommend just hitting the production turbopuffer service directly, but with a separate test org/API key: https://turbopuffer.com/docs/testing Works well for the vast majority of our customers (although we get the very occasional complaint about wanting a dev environment that works offline). The dataset sizes for local dev are usually so small that the cost rounds to free.

> although we get the very occasional complaint about wanting a dev environment that works offline It's only occasional because the people who care about dev environments that work offline are most likely to just skip you and move on. For actual developer experience, as well as a number of use cases like customers with security and privacy concerns, being able to host locally is essential. Fair enough if you don't ca…

Yep, we're well aware of the selection bias effects in product feedback. As we grow we're thinking about how to make our product more accessible to small orgs / hobby projects. Introducing a local dev environment may be part of that.

Note that we already have a in-your-own-VPC offering for large orgs with strict security/privacy/regulatory controls.

Re: ANN v3: 200ms p99 query latency over 100B vectors

#42

Post by benesch »

sa-code wrote: post #36
benesch wrote: post #34

Earlier quoted context omitted.

So I can note this down on our roadmap, what's the root of your requirement here? Supporting local dev without internet (airplanes, coffee shops, etc.)? Unit test speed? Something else?

I listed some reasons in another comment: https://news.ycombinator.com/item?id=46757853 I appreciate your responsiveness and open mind

Thanks, appreciate this! Jotted down some notes on our roadmap.

Re: ANN v3: 200ms p99 query latency over 100B vectors

#43

Post by sa-code »

benesch wrote: post #42
sa-code wrote: post #36

Earlier quoted context omitted.

I listed some reasons in another comment: https://news.ycombinator.com/item?id=46757853 I appreciate your responsiveness and open mind

Thanks, appreciate this! Jotted down some notes on our roadmap.

I wish you the best

Re: ANN v3: 200ms p99 query latency over 100B vectors

#44

Post by benesch »

dzbarsky wrote: post #39
benesch wrote: post #30

Earlier quoted context omitted.

My point is it's enough of a hassle to set up that I've yet to see that level of restriction in practice (across hundreds of CI systems).

Look into Bazel, a very standard build system used at many large tech companies. It splits fetches from build/test actions and allows blocking network for build/test actions with a single CLI flag. No hassle at all. The fact that you haven't come across this kind of setup suggests that your hundreds of CI systems are not representative of the industry as a whole.

I agree our sample may not be representative but we try to stay focused on the current and next crop of tpuf customers rather than the software industry as a whole. So far "CI prohibits network access during tests" just hasn't come up as a pain point for any of them, but as I mentioned in another comment [0], we're definitely keeping an open mind about introducing an offline dev experience.

(I am familiar with Bazel, but I'll have to save the war stories for another thread. It's not a build tool we see our particular customers using.)

[0]: https://news.ycombinator.com/item?id=46758156

Re: ANN v3: 200ms p99 query latency over 100B vectors

#45

Post by nostrebored »

sa-code wrote: post #32
nostrebored wrote: post #19

Earlier quoted context omitted.

Qdrant is one of the few vendors I actively steer people away from. Look at the GitHub issues, look at what their CEO says, look at their fake “advancements” that they pay for publicity on… The number of people I know who’ve had unrecoverable shard failures on Qdrant is too high to take it seriously.

I’m curious about this. Could you please point to some things the CEO has said, or reports of shard failures? The bit about paying for publicity doesn’t bother me. Edit: I haven’t found anything egregious that the CEO has said, or anything really sketchy. The shard failure warnings look serious, but the issues look closed https://github.com/qdrant/qdrant/issues/6025 https://github.com/qdrant/qdrant/issues/4939

https://x.com/nils_reimers/status/1809334134088622217?s=46

https://x.com/generall931/status/1809303448837582850?s=46

There used to be a benchmarking issue with a founder that was particularly egregious but I can’t find it anymore.

The sharding and consensus issues were from around a year and a half ago, so maybe it’s gotten better.

There are just so many options in the space, I don’t know why you’d go with one of the least correct vendors (whether or not the correctness is deception is a different question that I can’t answer)

Re: ANN v3: 200ms p99 query latency over 100B vectors

#46

Post by vander_elst »

benesch wrote: post #38
vander_elst wrote: post #33

> 504MiB shared L3 cache What CPU are they using here?

The exact CPU depends on the region/cloud provider, but this Granite Rapids CPU is representative: https://www.intel.com/content/www/us/en/products/sku/240777/...

Thanks!

Re: ANN v3: 200ms p99 query latency over 100B vectors

#47

Post by andre-z »

nostrebored wrote: post #19
sa-code wrote: post #10

Earlier quoted context omitted.

Qdrant is also a good default choice, since it can work in-memory for development, with a hard drive for small deployments and also for "web scale" workloads. As a principal eng, side-stepping a migration and having a good local dev experience is too good of a deal to pass up. That being said, turbopuffer looks interesting. I will check it out. Hopefully their local dev experience is good

Qdrant is one of the few vendors I actively steer people away from. Look at the GitHub issues, look at what their CEO says, look at their fake “advancements” that they pay for publicity on… The number of people I know who’ve had unrecoverable shard failures on Qdrant is too high to take it seriously.

What do I say? Happy to talk about "fakes". Here is my calendar. Feel free to book a slot. https://qdrant.to/andre-z

Re: ANN v3: 200ms p99 query latency over 100B vectors

#48

Post by enigmo »

benesch wrote: post #27
enigmo wrote: post #25

Earlier quoted context omitted.

having a local simulator (DynamoDB, Spanner, others) helps me a lot for offline/local development and CI. when a vendor doesn't off this I have often end up mocking it out (one way or another) and have to wait for integration or e2e tests for feedback that could have been pushed further to the left. in many CI environments unit tests don't have network access, it's not purely a price consideration. (not a turbopuffer…

> in many CI environments unit tests don't have network access, it's not purely a price consideration. I've never seen a hard block on network access (how do you install packages/pull images?) but I am sympathetic to wanting to enforce that unit tests run quickly by minimizing/eliminating RTT to networked services. We've considered the possibility of a local simulator before. Let me know if it winds up being a blocke…

you pull packages from a trusted package repository, not from the internet. this is not rare in my experience (financial services, security) and will become increasingly common due to software supply chain issues.

Re: ANN v3: 200ms p99 query latency over 100B vectors

#49

Post by bobvanluijt »

nostrebored wrote: post #45
sa-code wrote: post #32

Earlier quoted context omitted.

I’m curious about this. Could you please point to some things the CEO has said, or reports of shard failures? The bit about paying for publicity doesn’t bother me. Edit: I haven’t found anything egregious that the CEO has said, or anything really sketchy. The shard failure warnings look serious, but the issues look closed https://github.com/qdrant/qdrant/issues/6025 https://github.com/qdrant/qdrant/issues/4939

https://x.com/nils_reimers/status/1809334134088622217?s=46 https://x.com/generall931/status/1809303448837582850?s=46 There used to be a benchmarking issue with a founder that was particularly egregious but I can’t find it anymore. The sharding and consensus issues were from around a year and a half ago, so maybe it’s gotten better. There are just so many options in the space, I don’t know why you’d go with one of the…

> issue with a founder

That would be me

Post reply on HN