What does heterogeneous mean here?
HSE: Heterogeneous-memory storage engine designed for SSDs
31–40 of 47 posts
Re: HSE: Heterogeneous-memory storage engine designed for SSDs
#32What does heterogeneous mean here?
Multiple types of storage at the same time. For example, two different types of SSD, or combinations of SSD and DRAM.
Re: HSE: Heterogeneous-memory storage engine designed for SSDs
#33"World's first" Open-Source storage engine for SSDs? I believe Aerospike has advertised itself as that for years, and certainly most MongoDB instances are backed by SSD these days. Heck, conceptually etcd is a key-value storage engine built for SSDs. > HSE optimizes performance and endurance by orchestrating data placement across DRAM and multiple classes of SSDs or other solid-state storage. Orchestrating data place…
Isn't this 'HSE' conceptually a HSM? How does it compares to existing field-proven 'storage engines', some of them shock-full of features (because they are filesystems), such as Lustre or ZFS?
https://en.wikipedia.org/wiki/Hierarchical_storage_managemen...
https://en.wikipedia.org/wiki/ZFS#Caching_mechanisms:_ARC,_L...
Re: HSE: Heterogeneous-memory storage engine designed for SSDs
#34Looks pretty cool when you make it to the GitHub ( https://github.com/hse-project ). Order of magnitude performance gains! I imagine most of that come from skipping the Filesystem layer and just hitting the raw Block layer directly. I am curious about the durability and how well tested all of that is though. On the one hand, filesystems put a lot of work towards ensuring that bytes written to disk and synced are most…
SSDs are already raid devices internally so they're really no point as whatever you can do the vendor can and do faster and better in hardware/firmware.
That's why all the hyperscalers all have their own custom SKUs.
Re: HSE: Heterogeneous-memory storage engine designed for SSDs
#35Someone needs to write a book about breaking into writing software like RocksDB, HSE, etc. Years ago I found myself wanting to learn more however going from 0 to 1 felt impossible. Graduated from a T3 school in CS so understanding the concepts wasn't the issue, I just didn't know how to build a good foundation in low latency persistence. Years later I ended up contributing to low latency java which was really interes…
I just finished reading the OSTEP book[1] and it has a nice chapter on SSDs[2]. The entire last portion of the book is about filesystems/disks so you might find it interesting. [1] http://pages.cs.wisc.edu/~remzi/OSTEP/ [2] http://pages.cs.wisc.edu/~remzi/OSTEP/file-ssd.pdf
Oh my god this book is amazing! Thank you
Re: HSE: Heterogeneous-memory storage engine designed for SSDs
#36Will it perform well on all SSD's, or is it optimized to give top performance only on Micron devices?
Re: HSE: Heterogeneous-memory storage engine designed for SSDs
#37Someone needs to write a book about breaking into writing software like RocksDB, HSE, etc. Years ago I found myself wanting to learn more however going from 0 to 1 felt impossible. Graduated from a T3 school in CS so understanding the concepts wasn't the issue, I just didn't know how to build a good foundation in low latency persistence. Years later I ended up contributing to low latency java which was really interes…
"How can I learn an instrument without listening to music" "How can I learn to write stories without reading books?" Have you considered just reading the code? It's all available. Best way to learn is to look what the masters are doing. And even complex software is typically just a collection of simple things together. IMHO one of the biggest failing of many CS courses is that they never get above toy software. Would…
Reading the code of a piece of production software is a poor way to get understanding of the fundamentals of a strange topic. It's really inefficient in terms of "understanding" per hour; the student has no context as to why certain choices were made for what reason, just that they were made.
Re: HSE: Heterogeneous-memory storage engine designed for SSDs
#38> https://github.com/hse-project/hse Their benchmarks show significant gains compared to RocksDB. > https://github.com/spdk/rocksdb But what I'd really like to see is a comparison against RocksDB using SPDK > https://dqtibwqq6s6ux.cloudfront.net/download/papers/Hitachi... Based on these results, SPDK performs significantly better than the kernel requiring only 1-2 cores to saturate IOPS on an NVMe SSD (compared to th…
SPDK has had its share of detractors here on news.yc [0], especially with io_uring around the block [1]. It'd be interesting to see the improvements to these io-centric applications once they move to io_uring [2], which, in a way, like RocksDB, is sponsored by Facebook [3]. [0] https://news.ycombinator.com/item?id=10511960 [1] https://news.ycombinator.com/item?id=22266503 [2] https://news.ycombinator.com/item?id=1984…
For those who is wondering about what is io_uring and its significance I'd highly recommend this blog article posted at HN few days ago[1].
The potential of using io_uring with eBPF in Linux will makes any HPC enthusiast drooling :-)
Re: HSE: Heterogeneous-memory storage engine designed for SSDs
#39HSE not to be confused with HSE: https://www.hse.gov.uk/
Re: HSE: Heterogeneous-memory storage engine designed for SSDs
#40Someone needs to write a book about breaking into writing software like RocksDB, HSE, etc. Years ago I found myself wanting to learn more however going from 0 to 1 felt impossible. Graduated from a T3 school in CS so understanding the concepts wasn't the issue, I just didn't know how to build a good foundation in low latency persistence. Years later I ended up contributing to low latency java which was really interes…
I just finished reading the OSTEP book[1] and it has a nice chapter on SSDs[2]. The entire last portion of the book is about filesystems/disks so you might find it interesting. [1] http://pages.cs.wisc.edu/~remzi/OSTEP/ [2] http://pages.cs.wisc.edu/~remzi/OSTEP/file-ssd.pdf
That chapter on SSDs looks pretty good to me. Their numbers for NAND page and especially erase block sizes are very outdated; more modern values are 4kB to 16kB for NAND pages and 16-24MB for erase blocks on TLC NAND. Section 44.9 on mapping table sizes is a little bit odd, because most SSDs really do have 1GB of RAM per 1TB of flash, and that expense is widely seen as worthwhile even for multi-TB SSDs. The exceptions are low-end consumer SSDs that cache only part of the mapping table in a smaller amount of DRAM or SRAM, and a few enterprise/datacenter models that use 32kB block sizes for their FTL instead of the typical 4kB and thus reduce the DRAM requirement by a factor of 8 at the expense of greatly lowered performance and increased write amplification when writing in units smaller than 32kB.
Aside from the two above issues, everything looks correct and relevant, and I can't think of any missing details that deserve to be added to an introduction of that length.