21–30 of 74 posts
Utilizing: https://memcached.org/blog/nvm-caching/,https://github.com/m...
TLDR; Grafana Cloud needed tons of Caching, and it was expensive. So they used extstore in memcache to hold most of it on NVMe disks. This massively reduced their costs.
Interesting article here: https://grafana.com/blog/2023/08/23/how-we-scaled-grafana-cl... Utilizing: https://memcached.org/blog/nvm-caching/,https://github.com/m... TLDR; Grafana Cloud needed tons of Caching, and it was expensive. So they used extstore in memcache to hold most of it on NVMe disks. This massively reduced their costs.
I don't remember exactly why I have any of them saved, but these are some experimental data stores that seems to be fitting what you're looking for somewhat: - https://github.com/DataManagementLab/ScaleStore - "A Fast and Cost-Efficient Storage Engine using DRAM, NVMe, and RDMA" - https://github.com/unum-cloud/udisk ( https://github.com/unum-cloud/ustore ) - "The fastest ACID-transactional persisted Key-Value store d…
Earlier quoted context omitted.
Latency ought to be much better, since you're skipping several abstraction layers in the kernel. But that's about it. And the latency is still worse than in-memory solutions. Between that and the non-trivial effort needed to make this work in any sort of cloud setup (be it self-hosted k8s or AWS), it's a hard sell. If I really need latency above all, AWS gives me instances with 24TB RAM, and if I don't… why not just…
Agreed. The classic reason is when you have latency needs, but your data set is large enough that RAM is cost-prohibitive, and random-access enough that disk won’t work. The cost savings from switching to NVMe have to justify the higher NRE cost, and simultaneously, you have to be sensitive to latency.
Naive question: are there really expected gains to address natively an NVMe disk wrt using a regular key-value database on a filesystem ?
I believe that NVMe uses multiple I/O queues compared to serialized access with SATA and I think you’d be able to side unnecessary abstractions like file systems and block-based access with an NVMe-specific datastore. I’m also curious if different and more performant data structures can leveraged; if so, there may be downstream improvements for garbage collection, retrieval, and request parallelism.
The exact semantics vary per protocol but it’s a feature of most protocols at least in the currently used revisions: https://en.wikipedia.org/wiki/Native_Command_Queuing
https://github.com/OpenMPDK/KVRocks Given however, that most of the world has shifted to VMs, I don't think KV storage is accessible for that reason alone because the disks are often split out to multiple users. So the overall demand for this would be low.
NVME's allow namespaces to be made - effectively letting multiple users all share an NVME device without interfering with each other.
Some u.2 drives even support thin provisioning, like how a hypervisor treats a sparse disk file but for physical hardware.
I don't remember exactly why I have any of them saved, but these are some experimental data stores that seems to be fitting what you're looking for somewhat: - https://github.com/DataManagementLab/ScaleStore - "A Fast and Cost-Efficient Storage Engine using DRAM, NVMe, and RDMA" - https://github.com/unum-cloud/udisk ( https://github.com/unum-cloud/ustore ) - "The fastest ACID-transactional persisted Key-Value store d…
SolidCache and SolidQueue from Rails will be doing that when released. Otherwise though…you have the file system. Is that not enough?
Is that discussion/implementation of nvme available somewhere in public? https://github.com/rails/solid_cache didn't include anything about NVME that I could find.
So Solid Cache and Solid Queue just use the database (MySQL), which uses NVMe.
So now, in addition to: "You don't need a queue, just use Postgres/MySQL", we have "You don't need a cache, just use Postgres/MySQL"
I don't remember exactly why I have any of them saved, but these are some experimental data stores that seems to be fitting what you're looking for somewhat: - https://github.com/DataManagementLab/ScaleStore - "A Fast and Cost-Efficient Storage Engine using DRAM, NVMe, and RDMA" - https://github.com/unum-cloud/udisk ( https://github.com/unum-cloud/ustore ) - "The fastest ACID-transactional persisted Key-Value store d…
you could also configure Redis to transact everything to disk and choose nvme as the target
Good overview: https://www.mydistributed.systems/2020/07/towards-building-h...
The Azure Lv3/Lsv3/Lav3/Lasv3 series all provide this capability, for example.
Ref: https://learn.microsoft.com/en-us/azure/virtual-machines/las...