Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modern storage is plenty fast, but the APIs are bad

itnext.io

Post reply on HN
Print view View original

111–120 of 168 posts

Re: Modern storage is plenty fast, but the APIs are bad

#111

Post by magicalhippo »

mistrial9 wrote: post #101
echlebek wrote: post #98

Earlier quoted context omitted.

Consumer NVMe devices can deliver GB/s I/O and hundreds of thousands of iops. The article's point doesn't hinge on Optane at all.

please ask my several NVMe devices to take notice ! actual performance under Linux OS is far less than that, here

Single-thread, single-queue performance is much lower than the max with good NVMe devices.

With increased concurrency and deeper queues, my Samsung 960 Pro which has been running my Windows 10 desktop for several years still can do 294k random 4k reads IOPS, and 2.5GB/s sequential read.

Re: Modern storage is plenty fast, but the APIs are bad

#112

Post by smcleod »

rtkwe wrote: post #57
mehrdadn wrote: post #32

I liked most of the piece, but some bits rubbed me the wrong way: > I was taken by surprise by the fact that although every one of my peers is certainly extremely bright, most of them carried misconceptions about how to best exploit the performance of modern storage technology leading to suboptimal designs, even if they were aware of the increasing improvements in storage technology. > In the process of writing this…

Yeah how many people are running apps on servers served at all or even partially by NVMe SSDs? Where I work for our on prem stuff it's basically all network storage.

You can do pretty amazing things with well designed (onprem) networked storage with NVMe drives/arrays.

I replaced the company I was working with at the time's traditional "enterprise" HPE SANs with standard linux servers running a mix of NVMe and SATA SSDs that provided highly available, low latency and decent throughput iSCSI via network.

Gen 1 back in 2014/2015 did something like 70K random 4k read/write IOP/s per VM (running on Xen back then) and would just keep scaling till you hit the clusters 4M~ IOP/s limit (minus some overhead obviously).

Gen 2 provided between 100K and 200K random 4k to each VM to a limit of about 8M~ on the underlying units (which again were very affordable and low maintenance).

This provided very good storage performance (latency, throughput and fast / minimally if at all disruptive fail-over and recovery) for our apps, some of them were written in highly blocking Python code and needed to be rewritten async to get the most out of it, but it made a _huge_ (business changing) difference and saved us an insane amount of money.

These days I've moved into consulting and all the work I do is on GCP and AWS but I do miss the hands on high performing gear like that.

Old stuff now but the links are https://www.dropbox.com/s/rdojhb399639e4k/lightning_san.pdf?... and https://smcleod.net/tech/2015/07/24/scsi-benchmarking/ and there's a few other now quite dated posts on there.

Re: Modern storage is plenty fast, but the APIs are bad

#113

Post by charrondev »

rtkwe wrote: post #57
mehrdadn wrote: post #32

I liked most of the piece, but some bits rubbed me the wrong way: > I was taken by surprise by the fact that although every one of my peers is certainly extremely bright, most of them carried misconceptions about how to best exploit the performance of modern storage technology leading to suboptimal designs, even if they were aware of the increasing improvements in storage technology. > In the process of writing this…

Yeah how many people are running apps on servers served at all or even partially by NVMe SSDs? Where I work for our on prem stuff it's basically all network storage.

We did a big upgrade upgrade a couple years ago moving all of our DBs onto NVMe SSDs. We get significant improvements to our query times.

Fast SSDs are pretty cheap nowadays.

Re: Modern storage is plenty fast, but the APIs are bad

#114

Post by mikepurvis »

tonyarkles wrote: post #106
mikepurvis wrote: post #97

Earlier quoted context omitted.

I think a lot of it depends what the machines are used for. I'm not actually the IT department, but I believe in my org, we started out with a SAN-backed high availability cluster, because the immediate initial need was getting basic infrastructure (wiki, source control, etc) off of a dedicated machine that was a single point of failure. But then down the road a different set of hosts were brought online that had fas…

I’m laughing a little bit because an old place I used to work had a similar setup. The SAN/NAS/whatever it was was pretty slow, provisioning VMs was slow, and as much as we argued that we didn’t need redundancy for a lot of our VMs (they were semi-disposable), the IT department refused to give us a fast non-redundant machine. And then one day the SAN blew up. Some kind of highly unlikely situation where more disks fa…

Ha, that's awesome. Yeah, for the limited amount of stuff I maintain, I really like the simple, single-file Ansible script— install these handful of packages, insert this config file, set up this systemd service, and you're done. I know it's a lot harder for larger, more complicated systems where there's a lot of configuration state that they're maintaining internal to themselves and they want you to be setting up in-band using a web gui.

I experienced this recently trying to get a Sentry 10 cluster going— it's now this giant docker-compose setup with like 20 containers, and so I'm like "perfect, I'll insert all the config in my container deployment setup and then this will be trivially reproducible." Nope, turns out the particular microservice that I was trying to configure only uses its dedicated config file for standalone/debugging purposes; when it's being launched as part of the master system, everything is passed in at runtime and can only be set up from within the main app. Le sigh.

Re: Modern storage is plenty fast, but the APIs are bad

#115

Post by simcop2387 »

rtkwe wrote: post #57
mehrdadn wrote: post #32

I liked most of the piece, but some bits rubbed me the wrong way: > I was taken by surprise by the fact that although every one of my peers is certainly extremely bright, most of them carried misconceptions about how to best exploit the performance of modern storage technology leading to suboptimal designs, even if they were aware of the increasing improvements in storage technology. > In the process of writing this…

Yeah how many people are running apps on servers served at all or even partially by NVMe SSDs? Where I work for our on prem stuff it's basically all network storage.

This is starting to change a bit because of things like the DPUs that companies are making. Basically it's an intelligent PCI-e network bridge that lets you emulate/share PCI-e devices on the host while the actual hardware (NVMe storage, GPU, etc.) is located elsewhere. This lets you reconfigure the host in software without having to physically change the hardware in the servers itself. It also lets you change the way you have things in the server rack since everything doesn't need to be able to physically fit into every other server case.

EDIT: informative article about DPUs, https://www.servethehome.com/what-is-a-dpu-a-data-processing...

Re: Modern storage is plenty fast, but the APIs are bad

#116

Post by manigandham »

blindm wrote: post #4

Also: Modern storage is plenty fast, but also not reliable for long term use. That is why I buy a new SSD every year and clone my current (worn out) SSD to the new one. I have several old SSDs that started to get unhealthy, well, according to my S.M.A.R.T utility that I used to check them. I could probably get away with using an SSD for another year, but will not risk the data loss. Anyone else do this?

Consumer SSDs have endurance ratings in TBW which is terabytes written over the lifespan. They're often in the 100s with some drives over 1000. The faster drives also use MLC or TLC which has lower latency, better endurance, and higher performance than the more higher capacity QLC.

For example the Samsung 1TB 970 PRO (not the 980 PRO) has a 1200TBW rating with a 5 year warranty. That's 1.2M gigabytes written or more than 600GB every day, and will usually handle far more.

Re: Modern storage is plenty fast, but the APIs are bad

#117

Post by pjmlp »

justicezyx wrote: post #108
pjmlp wrote: post #103

Earlier quoted context omitted.

The problem is forcing everyone to program in asynchronous way. WinRT tried to go down that route (only asynchronous APIs), to drive developers into that path, but eventually they had to support synchronous as well due to the received resistence.

Before Herb Sutter's free lunch is over, not many writes in multi-threading code. Then today, everyone is writing multi-threading code, one way or another; and majority indirectly in newer languages like Go and Rust, or better frameworks like Actors, message passing, coroutines, and some noble souls who are capable enough, in classic pthread and other threading APIs. Of course everyone is going to program in an async…

Agreed, it is also why Microsoft was one of the biggest contributors to the co-routines support in C++, and have frameworks like Orleans and Coyote.

Re: Modern storage is plenty fast, but the APIs are bad

#118

Post by p1necone »

wtallis wrote: post #71
im3w1l wrote: post #59

Earlier quoted context omitted.

My mobo has many more SATA slots than M.2. slots. I expect there will be hybrid systems for quite a while.

One SSD is sufficient for almost all consumer systems. The only reason to want more than two SSDs is if you're re-using at least one old tiny SSD in a new machine. SATA ports will stick around in desktops only for the sake of hard drives. There may be a few niches left where using several SATA SSDs in a workstation still makes some kind of sense, and obviously not all server platforms have migrated to NVMe yet. But a…

Drive price doesn't scale linearly with capacity, you can save a fair bit of money sticking with multiple smaller capacity drives vs one big one.

Re: Modern storage is plenty fast, but the APIs are bad

#119

Post by throwaway_pdp09 »

_urga wrote: post #65
throwaway_pdp09 wrote: post #48

Earlier quoted context omitted.

> capped by the line fill buffer and queuing delay kicks in spiking your cache miss could you point me to a little reading material on this? I know what an LFB is, more or less, but what queueing delay, an dhow does that relate to cache misses? Thanks.

Sure, I'm still pretty fuzzy on these things, but queueing delay is Little's law: https://en.wikipedia.org/wiki/Little's_law It means if a system can only do X of something per second, then if you push the system past that, new arriving stuff has to wait on existing work in the queue, and things take longer than if the queue was empty. You can think of it like a traffic jam and it applies to most systems. For example…

Appreciated, thanks

Re: Modern storage is plenty fast, but the APIs are bad

#120

Post by eightysixfour »

fibers wrote: post #88
eightysixfour wrote: post #79

Earlier quoted context omitted.

Yes, using PCIe expansion cards. I know of an AMD board that ships with 5 (3 on the board, 2 with a PCIe card). Could easily add more.

is this with threadripper boards?

No, just an x570, the MSI “Godlike”. You can also just buy PCIe cards with M2 slots for drives.
Post reply on HN