Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why are there no open source NVMe-native key value stores in 2023?

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 74 posts

Re: Ask HN: Why are there no open source NVMe-native key value stores in 2023?

#11

Post by formerly_proven »

There's actually an NVMe command set which allows you to use the FTL directly as a K/V store. (This is limited to 16-byte keys [1] however, so it is not that useful and probably not implemented anywhere, my guess is Samsung looked at this for some hyperscaler, whipped up a prototype in their customer-specific firmware and the benefits were lesser than expected so it's dead now)

[1] These slides claim up to 32 bytes, which would be a practically useful length: https://www.snia.org/sites/default/files/ESF/Key-Value-Stora... but the current revision of the standard only permits two 64-bit words as the key ("The maximum KV key size is 16 bytes"): https://nvmexpress.org/wp-content/uploads/NVM-Express-Key-Va...

Re: Ask HN: Why are there no open source NVMe-native key value stores in 2023?

#13

Post by londons_explore »

chaos_emergent wrote: post #9
bestouff wrote: post #4

Naive question: are there really expected gains to address natively an NVMe disk wrt using a regular key-value database on a filesystem ?

I believe that NVMe uses multiple I/O queues compared to serialized access with SATA and I think you’d be able to side unnecessary abstractions like file systems and block-based access with an NVMe-specific datastore. I’m also curious if different and more performant data structures can leveraged; if so, there may be downstream improvements for garbage collection, retrieval, and request parallelism.

Most filesystems will make use of multiple IO queues - ie. if an application sends many different read requests, they may be satisfied out-of-order.

Re: Ask HN: Why are there no open source NVMe-native key value stores in 2023?

#14

Post by londons_explore »

formerly_proven wrote: post #11

There's actually an NVMe command set which allows you to use the FTL directly as a K/V store. (This is limited to 16-byte keys [1] however, so it is not that useful and probably not implemented anywhere, my guess is Samsung looked at this for some hyperscaler, whipped up a prototype in their customer-specific firmware and the benefits were lesser than expected so it's dead now) [1] These slides claim up to 32 bytes,…

Presumably there is some way to use the hash of the actual key as the key, and then store both key and value as data?

16 bytes is long enough that collisions will be super rare, and while you obviously need to write code to support that case, it should have no performance impact.

Re: Ask HN: Why are there no open source NVMe-native key value stores in 2023?

#15

Post by londons_explore »

delfinom wrote: post #5

https://github.com/OpenMPDK/KVRocks Given however, that most of the world has shifted to VMs, I don't think KV storage is accessible for that reason alone because the disks are often split out to multiple users. So the overall demand for this would be low.

NVME's allow namespaces to be made - effectively letting multiple users all share an NVME device without interfering with each other.

Re: Ask HN: Why are there no open source NVMe-native key value stores in 2023?

#16

Post by londons_explore »

formerly_proven wrote: post #11

There's actually an NVMe command set which allows you to use the FTL directly as a K/V store. (This is limited to 16-byte keys [1] however, so it is not that useful and probably not implemented anywhere, my guess is Samsung looked at this for some hyperscaler, whipped up a prototype in their customer-specific firmware and the benefits were lesser than expected so it's dead now) [1] These slides claim up to 32 bytes,…

I could imagine that if this mode isn't widely used, drive manufacturers haven't given much thought to performance, and it therefore might suck.

Re: Ask HN: Why are there no open source NVMe-native key value stores in 2023?

#17

Post by londons_explore »

formerly_proven wrote: post #11

There's actually an NVMe command set which allows you to use the FTL directly as a K/V store. (This is limited to 16-byte keys [1] however, so it is not that useful and probably not implemented anywhere, my guess is Samsung looked at this for some hyperscaler, whipped up a prototype in their customer-specific firmware and the benefits were lesser than expected so it's dead now) [1] These slides claim up to 32 bytes,…

I think some devices built the block storage on top of the key-value store. Ie. when you write "hello..." (4k bytes) to address 123, it actually saves key: 123, value "hello...".

If so, that is probably the reason for a 16 byte key - there is just no way anybody needs a key bigger than 16 bytes for an address anytime soon.

Re: Ask HN: Why are there no open source NVMe-native key value stores in 2023?

#18

Post by klodolph »

creshal wrote: post #7
bestouff wrote: post #4

Naive question: are there really expected gains to address natively an NVMe disk wrt using a regular key-value database on a filesystem ?

Latency ought to be much better, since you're skipping several abstraction layers in the kernel. But that's about it. And the latency is still worse than in-memory solutions. Between that and the non-trivial effort needed to make this work in any sort of cloud setup (be it self-hosted k8s or AWS), it's a hard sell. If I really need latency above all, AWS gives me instances with 24TB RAM, and if I don't… why not just…

Agreed. The classic reason is when you have latency needs, but your data set is large enough that RAM is cost-prohibitive, and random-access enough that disk won’t work. The cost savings from switching to NVMe have to justify the higher NRE cost, and simultaneously, you have to be sensitive to latency.

Re: Ask HN: Why are there no open source NVMe-native key value stores in 2023?

#19

Post by andruby »

brightball wrote: post #8

SolidCache and SolidQueue from Rails will be doing that when released. Otherwise though…you have the file system. Is that not enough?

Is that discussion/implementation of nvme available somewhere in public?

https://github.com/rails/solid_cache didn't include anything about NVME that I could find.

Re: Ask HN: Why are there no open source NVMe-native key value stores in 2023?

#20

Post by di4na »

bestouff wrote: post #4

Naive question: are there really expected gains to address natively an NVMe disk wrt using a regular key-value database on a filesystem ?

Yes, mostly on the durability side. NVMe actually has the relevant API to be sure that a write was flushed, while posix like filesystem API usually do not handle it.
Post reply on HN