Live data from Hacker News

Ask HN: Why are there no open source NVMe-native key value stores in 2023?

news.ycombinator.com

71–74 of 74 posts

Re: Ask HN: Why are there no open source NVMe-native key value stores in 2023?

#71
post #7

Earlier quoted context omitted.

Latency ought to be much better, since you're skipping several abstraction layers in the kernel. But that's about it. And the latency is still worse than in-memory solutions. Between that and the non-trivial effort needed to make this work in any sort of cloud setup (be it self-hosted k8s or AWS), it's a hard sell. If I really need latency above all, AWS gives me instances with 24TB RAM, and if I don't… why not just…

While you can get 24TB ram, there is a pretty big cost difference. 2 TB of ram costs roughly $10000 compared to $130 for NVME storage (or $230 for 12 TB of a good hard drive). Sure the NVME is ~3.5x more expensive, but the latency will be dramatically lower and the throughput will be dramatically higher. Sure you can build a 24 TB ram system, but at that point the cost of the server will be entirely the ram. The reas…

All that applies to regular kv stores abstracted through filesystems and block device layers just fine.

But when your latency requirements are so tight that you cannot possibly afford the latency penalty of a filesystem, you better have a good business case to justify either developing a custom bare-metal-nvme (which is $$$$$ and takes time) or getting a multi-TB RAM system, which is also $$$$$, but far more predictable, and can be put into production today, not 6+ months later when you finish developing your custom kv store.

For the other 99.999% of use cases, sure, just go with NVMe backing your regular virtualization/containerization infrastructure.

Re: Ask HN: Why are there no open source NVMe-native key value stores in 2023?

#72
I work on a database that is a KV-store if you squint enough and we're taking advantage of NVMe.

One thing they don't tell you about NVMe is you'll end up bottlenecked on CPU and memory bandwidth if you do it right. The problem is after eliminating all of the speed bumps in your IO pathway, you have a vertical performance mountain face to climb. People are just starting to run into these problems, so it's hard to say what the future holds. It's all very exciting.

Re: Ask HN: Why are there no open source NVMe-native key value stores in 2023?

#73
post #36

Earlier quoted context omitted.

See https://www.snia.org/educational-library/key-value-standardi... for some description of the special command set to get an nvme drive to natively work as a key-value store. Also https://www.snia.org/sites/default/files/ESF/Key-Value-Stora...

How is that implemented? Btree, hashtable?

Implementation-defined. The API resembles a map access, though.

Re: Ask HN: Why are there no open source NVMe-native key value stores in 2023?

#74

Earlier quoted context omitted.

See https://www.snia.org/educational-library/key-value-standardi... for some description of the special command set to get an nvme drive to natively work as a key-value store. Also https://www.snia.org/sites/default/files/ESF/Key-Value-Stora...

How do you tell which NVMe drive models support the KV API? Is this something that you can experiment with on a consumer drive or do you need specific enterprise ssd models? Samsung's uNVMe evaluation guide (from 2019) device support section just states: Guide Version: uNVMe2.0 SDK Evaluation Guide ver 1.2 Supported Product(s): NVMe SSD (Block/KV) Interface(s): NVMe 1.2 https://github.com/OpenMPDK/uNVMe/blob/master/d…

I reached out to Samsung support to ask. After being sent from one department to another and receiving some very clearly incorrect advice from their sales support they eventually sent me to an online form for the memory department. Am still waiting for a response a week later.
Post reply on HN