Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modern storage is plenty fast, but the APIs are bad

itnext.io

Post reply on HN
Print view View original

41–50 of 168 posts

Re: Modern storage is plenty fast, but the APIs are bad

#41

Post by rcxdude »

blindm wrote: post #4

Also: Modern storage is plenty fast, but also not reliable for long term use. That is why I buy a new SSD every year and clone my current (worn out) SSD to the new one. I have several old SSDs that started to get unhealthy, well, according to my S.M.A.R.T utility that I used to check them. I could probably get away with using an SSD for another year, but will not risk the data loss. Anyone else do this?

I've had one (very early and cheap) SSD fail on me. Other than that I don't think I've seen or heard of any issues across a large range of more modern SSDs. The reliability and endurange issues which occured on earlier SSDs no longer seem to be a problem (this is in part because flash density has skyrocketed: because each flash chip can operate more or less independently, the more storage an SSD has the faster it can run and the more write endurance it has).

Re: Modern storage is plenty fast, but the APIs are bad

#42

Post by wtallis »

Blahah wrote: post #33
zbrozek wrote: post #11

Earlier quoted context omitted.

What do you do that wears them out so fast? I've been running the same NVMe disk as my daily driver since 2015 and it's not showing any signs of degradation.

I work with bioinformatics data and tend to switch out an NVMe within 3-4 months. I'm usually maxing out read or write for 12 out of 24 hours a day. The slowdown is rapid and very noticeable.

> The slowdown is rapid and very noticeable.

That probably doesn't have anything to do with write endurance of the flash memory. When your drive's flash is mostly worn-out, you will see latency affected as the drive has to retry reads and use more complex error correction schemes to recover your data. But there are several other mechanisms by which a SSD's performance will degrade early in its lifetime depending on the workload. Those performance degradations are avoidable to some extent, and are not permanent.

Re: Modern storage is plenty fast, but the APIs are bad

#43

Post by Blahah »

wtallis wrote: post #42
Blahah wrote: post #33

Earlier quoted context omitted.

I work with bioinformatics data and tend to switch out an NVMe within 3-4 months. I'm usually maxing out read or write for 12 out of 24 hours a day. The slowdown is rapid and very noticeable.

> The slowdown is rapid and very noticeable. That probably doesn't have anything to do with write endurance of the flash memory. When your drive's flash is mostly worn-out, you will see latency affected as the drive has to retry reads and use more complex error correction schemes to recover your data. But there are several other mechanisms by which a SSD's performance will degrade early in its lifetime depending on t…

So I can potentially recycle my used SSDs?

Re: Modern storage is plenty fast, but the APIs are bad

#44

Post by mehrdadn »

Ericson2314 wrote: post #37

From the author's previous piece: https://www.scylladb.com/2020/05/05/how-io_uring-and-ebpf-wi... > Our CTO, Avi Kivity, made the case for async at the Core C++ 2019 event. The bottom line is this; in modern multicore, multi-CPU devices, the CPU itself is now basically a network, the intercommunication between all the CPUs is another network, and calls to disk I/O are effectively another. There are good reasons why n…

> in modern multicore, multi-CPU devices, the CPU itself is now basically a network, the intercommunication between all the CPUs is another network, and calls to disk I/O are effectively another.

Interesting take, and NUMA CPUs have felt networked to me when I've used them, but typical multicore UMA CPUs sure haven't... is there a reason to believe this will change (or already has), or did the author mean to only talk about NUMA?

Re: Modern storage is plenty fast, but the APIs are bad

#47

Post by mehrdadn »

Blahah wrote: post #43
wtallis wrote: post #42

Earlier quoted context omitted.

> The slowdown is rapid and very noticeable. That probably doesn't have anything to do with write endurance of the flash memory. When your drive's flash is mostly worn-out, you will see latency affected as the drive has to retry reads and use more complex error correction schemes to recover your data. But there are several other mechanisms by which a SSD's performance will degrade early in its lifetime depending on t…

So I can potentially recycle my used SSDs?

I think you could give it a shot with ATA Secure Erasing one of them and seeing if it performs faster. Although 4 months at 50% utilization at (say) 2GB/s is some ~10PB of I/O, so I'm not sure if I would expect what you're seeing to be a temporary slowdown...

Re: Modern storage is plenty fast, but the APIs are bad

#48

Post by throwaway_pdp09 »

_urga wrote: post #14
ivoras wrote: post #2

> ...misconceptions... Yet if you skim through specs of modern NVMe devices you see commodity devices with latencies in the microseconds range and several GB/s of throughput supporting several hundred thousands random IOPS. So where’s the disconnect? Whoa there... let's not compare devices with 20+ GB/s and latencies in nanosecond ranges which translate to half a dozen giga-ops per second (aka RAM) with any kind of f…

The advertised bandwidth for RAM is not actually what you get per-core, which is what you care about in practice. If you want to know the upper bound on your per-core RAM bandwidth: 64 bytes (the size of a cache line) * 10 slots (in a CPU core's LFB or line fill buffer) / 100ns (the typical cost of a cache miss) * 1000000 * 1000 (to convert ns to ms to seconds) = 6400000000 bytes per second = 5.96 GiB per second RAM…

> capped by the line fill buffer and queuing delay kicks in spiking your cache miss

could you point me to a little reading material on this? I know what an LFB is, more or less, but what queueing delay, an dhow does that relate to cache misses? Thanks.

Re: Modern storage is plenty fast, but the APIs are bad

#49

Post by mmis1000 »

blindm wrote: post #23
thekrendal wrote: post #16

Earlier quoted context omitted.

That's still nothing even if you do that 4x/day. Also just because you create a 20GB virtual disk does not necessarily mean you're actually writing out 20GB to the disk. Many SSDs and NVMEs are designed with total drive writes per day in their specs. What is the wear method you're measuring by and what's the threshold where you're replacing your drives?

> does not necessarily mean you're actually writing out 20GB to the disk. You mean like preallocation? I think Virtualbox now does that. In the past it didn't though, it just kept writing a bunch of zeroes to the drive until it reached 20GB.

Or probably the filesystem decides to do it? The Refs will just ate adjacent 0 and assuming you want a fallocate here.

Re: Modern storage is plenty fast, but the APIs are bad

#50

Post by CyberRabbi »

porpoise wrote: post #19
blindm wrote: post #4

Also: Modern storage is plenty fast, but also not reliable for long term use. That is why I buy a new SSD every year and clone my current (worn out) SSD to the new one. I have several old SSDs that started to get unhealthy, well, according to my S.M.A.R.T utility that I used to check them. I could probably get away with using an SSD for another year, but will not risk the data loss. Anyone else do this?

On the one hand, a new SSD a year sounds extreme. On the other hand, how many years does each of us have left? Ten? twenty? Thirty? Forty? Few of us can easily imagine ourselves still alive and productive in forty years. So much of what we do rests on an implicit assumption that we are going to live for eternity, and starts to seem pointless when we consider how short our existence is.

Materialism (the dominant underlying philosophy of our culture) keeps us away from that higher level consciousness. It poisons our mental models and worldview.
Post reply on HN