Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modern storage is plenty fast, but the APIs are bad

itnext.io

Post reply on HN
Print view View original

21–30 of 168 posts

Re: Modern storage is plenty fast, but the APIs are bad

#21

Post by smcameron »

gravypod wrote: post #3
ivoras wrote: post #2

> ...misconceptions... Yet if you skim through specs of modern NVMe devices you see commodity devices with latencies in the microseconds range and several GB/s of throughput supporting several hundred thousands random IOPS. So where’s the disconnect? Whoa there... let's not compare devices with 20+ GB/s and latencies in nanosecond ranges which translate to half a dozen giga-ops per second (aka RAM) with any kind of f…

Depending on the storage technology the comparison to RAM is not that far off. Intel is trying to market it that way any way [0]. It's obviously not RAM but it's not the [0] - https://www.intel.com/content/www/us/en/architecture-and-tec...

Yeah, back in 2014, I worked at HP on storage drivers for linux, and we got 1 million IOPS (4k random reads) on a single controller, with SSDs, but we had to do some fairly hairy stuff. This was back when NVME was new and we were trying to do SCSI over PCIe. We set up multiple ring buffers for command submission and command completion, one each per CPU, and pinned threads to CPUs and were very careful to avoid locking (e.g. spinlocks, etc.). I think we also had to pin some userland processes to particular CPUs to avoid NUMA induced bottlenecks.

The thing is, up until this point, for the entire history of computers, storage was so relatively slow compared to memory and the CPU that drivers could be quite simple, chuck requests and completions into queues managed by simple locking, and the fraction of time that requests spent inside the driver would still be negligible compared to the time they spent waiting for the disks. If you could theoretically make your driver infinitely fast, this would only amount to maybe a 1% speedup. So there was no need to spend a lot of time thinking about how to make the driver super efficient. Until suddenly there was.

Re: Modern storage is plenty fast, but the APIs are bad

#22

Post by gogopuppygogo »

wtallis wrote: post #5
ivoras wrote: post #2

> ...misconceptions... Yet if you skim through specs of modern NVMe devices you see commodity devices with latencies in the microseconds range and several GB/s of throughput supporting several hundred thousands random IOPS. So where’s the disconnect? Whoa there... let's not compare devices with 20+ GB/s and latencies in nanosecond ranges which translate to half a dozen giga-ops per second (aka RAM) with any kind of f…

The article isn't exactly conflating RAM and flash; if it were, the conclusions would be very different. A synchronous blocking IO API is fine if you're working with nanosecond latencies of RAM, or with storage that's as painfully slow and serial as a mechanical hard drive. Flash is special in that its latency is still considerably higher than that of DRAM, but its throughput can get reasonably close once you have mo…

Sustainable read/write speeds are also different than peak on SSD vs RAM.

Re: Modern storage is plenty fast, but the APIs are bad

#23

Post by blindm »

thekrendal wrote: post #16
blindm wrote: post #12

Earlier quoted context omitted.

> Every year seems like a very short lifespan Yes but I forgot to mention I do a lot of heavy writes to it. It is common to see me creating a huge 20GB virtual machine disk image, using it for a few hours, then deleting it, before creating a new one in its place. I'm a huge virtualization freak.

That's still nothing even if you do that 4x/day. Also just because you create a 20GB virtual disk does not necessarily mean you're actually writing out 20GB to the disk. Many SSDs and NVMEs are designed with total drive writes per day in their specs. What is the wear method you're measuring by and what's the threshold where you're replacing your drives?

> does not necessarily mean you're actually writing out 20GB to the disk.

You mean like preallocation? I think Virtualbox now does that. In the past it didn't though, it just kept writing a bunch of zeroes to the drive until it reached 20GB.

Re: Modern storage is plenty fast, but the APIs are bad

#24

Post by LandR »

blindm wrote: post #4

Also: Modern storage is plenty fast, but also not reliable for long term use. That is why I buy a new SSD every year and clone my current (worn out) SSD to the new one. I have several old SSDs that started to get unhealthy, well, according to my S.M.A.R.T utility that I used to check them. I could probably get away with using an SSD for another year, but will not risk the data loss. Anyone else do this?

I'm still on the SSD I bought 6 or 7 years ago as my OS drive.

Haven't noticed a single issue on it.

Re: Modern storage is plenty fast, but the APIs are bad

#25

Post by aszen »

porpoise wrote: post #19
blindm wrote: post #4

Also: Modern storage is plenty fast, but also not reliable for long term use. That is why I buy a new SSD every year and clone my current (worn out) SSD to the new one. I have several old SSDs that started to get unhealthy, well, according to my S.M.A.R.T utility that I used to check them. I could probably get away with using an SSD for another year, but will not risk the data loss. Anyone else do this?

On the one hand, a new SSD a year sounds extreme. On the other hand, how many years does each of us have left? Ten? twenty? Thirty? Forty? Few of us can easily imagine ourselves still alive and productive in forty years. So much of what we do rests on an implicit assumption that we are going to live for eternity, and starts to seem pointless when we consider how short our existence is.

Very well said, there are times we lose the bigger picture of our lives and instead start wasting times with pointless stuff just to escape the reality of our lives.

Re: Modern storage is plenty fast, but the APIs are bad

#26

Post by blindm »

Felk wrote: post #9
blindm wrote: post #4

Also: Modern storage is plenty fast, but also not reliable for long term use. That is why I buy a new SSD every year and clone my current (worn out) SSD to the new one. I have several old SSDs that started to get unhealthy, well, according to my S.M.A.R.T utility that I used to check them. I could probably get away with using an SSD for another year, but will not risk the data loss. Anyone else do this?

I think having a backup solution is the better choice here. You can use your SSDs until they die or become too slow, and you won't lose your data if it breaks before you replace it after a year

> I think having a backup solution is the better choice here

Any particular provider you would recommend? I've looked into backblaze but it seems a bit pricey. Also: I am aware that cloud based backup solutions have very little failure rate in terms of drives since they're probably using RAID

Re: Modern storage is plenty fast, but the APIs are bad

#27

Post by CyberRabbi »

blindm wrote: post #4

Also: Modern storage is plenty fast, but also not reliable for long term use. That is why I buy a new SSD every year and clone my current (worn out) SSD to the new one. I have several old SSDs that started to get unhealthy, well, according to my S.M.A.R.T utility that I used to check them. I could probably get away with using an SSD for another year, but will not risk the data loss. Anyone else do this?

When ssds fail they don’t lose your data, they just become unwritable. What you’re doing is unnecessary and wasteful.

Re: Modern storage is plenty fast, but the APIs are bad

#28

Post by NikolaeVarius »

blindm wrote: post #12
marcolussetti wrote: post #8

Earlier quoted context omitted.

Every year seems like a very short lifespan, but I guess every usecase is different. I definitely replace drive when SMART is starting to look bleak, but that is far more infrequent in my usecase I guess.

> Every year seems like a very short lifespan Yes but I forgot to mention I do a lot of heavy writes to it. It is common to see me creating a huge 20GB virtual machine disk image, using it for a few hours, then deleting it, before creating a new one in its place. I'm a huge virtualization freak.

That is absolutely nothing in terms of the write endurance for modern drives

Re: Modern storage is plenty fast, but the APIs are bad

#29

Post by jeroenhd »

I suppose _modern_ storage is fast, but how many servers are running on storage this modern? None of mine are and my work dev machine is still rocking a SATA 2.5" SSD.

We're probably still a few years off from being able to switch to this fast I/O yet. With the new game consoles switching over to PCIe SSDs I expect the price of NVMe drives to drop over the next few years until they're cheap enough that the majority of computers are running NVMe drives.

Even with SATA drives like mine though, there's really not that much performance loss from doing IO operations. I've run my OS with 8GiB of SSD swap in active use during debugging and while the stutters are annoying and distracting, the computer didn't grind to a halt like it would with spinning rust. Storage speed has increased massively in the last five years, for the love of god fellow developers, please make use of it when you can!

That said, deferring IO until you're done still makes sense for some consumer applications because cheap laptops are still being sold with hard drives and those devices are probably the minimum requirement you'll be serving.

Re: Modern storage is plenty fast, but the APIs are bad

#30

Post by smcameron »

smcameron wrote: post #21
gravypod wrote: post #3

Earlier quoted context omitted.

Depending on the storage technology the comparison to RAM is not that far off. Intel is trying to market it that way any way [0]. It's obviously not RAM but it's not the [0] - https://www.intel.com/content/www/us/en/architecture-and-tec...

Yeah, back in 2014, I worked at HP on storage drivers for linux, and we got 1 million IOPS (4k random reads) on a single controller, with SSDs, but we had to do some fairly hairy stuff. This was back when NVME was new and we were trying to do SCSI over PCIe. We set up multiple ring buffers for command submission and command completion, one each per CPU, and pinned threads to CPUs and were very careful to avoid lockin…

Oh yeah, iirc, the 1M IOPS driver was a block driver. For the SCSI over PCIe stuff, there was the big problem at the time that the entire SCSI layer in the kernel was a bottleneck, so you could make the driver as fast as you wanted, but your requests were still coming through a single queue managed by locks, so you were screwed. There was a whole ton of work done by Christoph Hellwig, Jens Axboe and others to make the SCSI layer "multiqueue" around that time to fix that.
Post reply on HN