HSE: Heterogeneous-memory storage engine designed for SSDs • Hacker News Board

Re: HSE: Heterogeneous-memory storage engine designed for SSDs

#11

Post by g14i » Mon, Apr 27, 2020, 9:55 PM UTC

Many techniques already used by Aerospike on their KV database, which also bypass the OS file system/cache.

I'm a long time Aerospike user with no connection to Aerospike.

Edit: I would love to see a benchmark with Aerospike.

Re: HSE: Heterogeneous-memory storage engine designed for SSDs

#12

Post by aloknnikhil » Mon, Apr 27, 2020, 10:03 PM UTC

> https://github.com/hse-project/hse

Their benchmarks show significant gains compared to RocksDB.

> https://github.com/spdk/rocksdb

But what I'd really like to see is a comparison against RocksDB using SPDK

> https://dqtibwqq6s6ux.cloudfront.net/download/papers/Hitachi... Based on these results, SPDK performs significantly better than the kernel requiring only 1-2 cores to saturate IOPS on an NVMe SSD (compared to the kernel requiring 16)

Re: HSE: Heterogeneous-memory storage engine designed for SSDs

#13

Post by wtallis » Mon, Apr 27, 2020, 10:07 PM UTC

aloknnikhil wrote: post #12

> https://github.com/hse-project/hse Their benchmarks show significant gains compared to RocksDB. > https://github.com/spdk/rocksdb But what I'd really like to see is a comparison against RocksDB using SPDK > https://dqtibwqq6s6ux.cloudfront.net/download/papers/Hitachi... Based on these results, SPDK performs significantly better than the kernel requiring only 1-2 cores to saturate IOPS on an NVMe SSD (compared to th…

I'd also like to see comparison to Toshiba/Kioxia's fork TRocksDB: https://github.com/KioxiaAmerica/trocksdb

Re: HSE: Heterogeneous-memory storage engine designed for SSDs

#14

Post by cryptonector » Mon, Apr 27, 2020, 10:12 PM UTC

Sounds kinda like a ZFS.

Re: HSE: Heterogeneous-memory storage engine designed for SSDs

#15

Post by jandrewrogers » Mon, Apr 27, 2020, 10:24 PM UTC

PR copy aside, the claimed performance differences relative to RocksDB and WiredTiger are typical of many storage engines, the performance doesn't stand out. I don't think either RocksDB or WT has made a serious claim to prioritizing performance in their designs in any case.

Also, I have to wonder how narrowly "open-source storage engine for SSDs" is being defined here such that it excludes so many earlier storage engines in claiming the title of "first".

Re: HSE: Heterogeneous-memory storage engine designed for SSDs

#16

Post by dang » Mon, Apr 27, 2020, 10:30 PM UTC

shockinglytrue wrote: post #6

Much better link: https://github.com/hse-project/hse PR is insane hot air referring to another hot air product (can you even buy their 3D Xpoint devices yet?)

Ok, we've changed the URL to that from https://investors.micron.com/news-releases/news-release-deta....

Re: HSE: Heterogeneous-memory storage engine designed for SSDs

#17

Post by drenvuk » Mon, Apr 27, 2020, 10:43 PM UTC

This is unbelievably cool. It is a multi segmented key prefix Key Value store. Can someone just strap paxos or raft to this and call it a day please? Pretty please?

Re: HSE: Heterogeneous-memory storage engine designed for SSDs

#18

Post by haivri » Mon, Apr 27, 2020, 10:49 PM UTC

drenvuk wrote: post #17

This is unbelievably cool. It is a multi segmented key prefix Key Value store. Can someone just strap paxos or raft to this and call it a day please? Pretty please?

I wonder how close of an API this provides compared to RocksDB... If close, CockroachDB might be a good trial candidate

Re: HSE: Heterogeneous-memory storage engine designed for SSDs

#19

Post by buildbot » Mon, Apr 27, 2020, 10:50 PM UTC

elihu wrote: post #9

So, is this open-source firmware that runs directly on Micron SSDs, or is it an upper-layer thing that runs on the host system?

I thought it was firmware too, but it appears to be more of a key value store block level access engine, and improves mongo performance.

I got really excited thinking it was an open source nvme fpga core.

Re: HSE: Heterogeneous-memory storage engine designed for SSDs

#20

Post by pinewurst » Mon, Apr 27, 2020, 11:35 PM UTC

fortran77 wrote: post #10

Will this be a replacement for something like the overpriced and under performing proprietary products from Pure Storage?

You're comparing apples to oranges, plus that's unduly harsh.

1. This is a KV store optimized (or claimed to be) for a combo of SSD and PMEM. Not a packaged, supported, appliance storage system.

2. People who pay for enterprise storage do so for reasons beyond being too bone-headed to appreciate the joys of cobbling together production systems from the white box low-bidder and open source software.