Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modern storage is plenty fast, but the APIs are bad

itnext.io

Post reply on HN
Print view View original

71–80 of 168 posts

Re: Modern storage is plenty fast, but the APIs are bad

#71

Post by wtallis »

im3w1l wrote: post #59
wtallis wrote: post #36

Earlier quoted context omitted.

> I expect the price of NVMe drives to drop over the next few years until they're cheap enough that the majority of computers are running NVMe drives. Price no longer has anything to do with it. PC OEMs are simply not shipping SATA SSDs any more, and major drive vendors have started to discontinue their client (OEM) SATA SSD product lines. We're just waiting for the SATA-based PC install base to be retired.

My mobo has many more SATA slots than M.2. slots. I expect there will be hybrid systems for quite a while.

One SSD is sufficient for almost all consumer systems. The only reason to want more than two SSDs is if you're re-using at least one old tiny SSD in a new machine. SATA ports will stick around in desktops only for the sake of hard drives. There may be a few niches left where using several SATA SSDs in a workstation still makes some kind of sense, and obviously not all server platforms have migrated to NVMe yet. But as far as influencing the direction and design of consumer systems, SATA SSDs have only slightly more relevance than optical disc drives.

Re: Modern storage is plenty fast, but the APIs are bad

#72

Post by digikata »

rtkwe wrote: post #57
mehrdadn wrote: post #32

I liked most of the piece, but some bits rubbed me the wrong way: > I was taken by surprise by the fact that although every one of my peers is certainly extremely bright, most of them carried misconceptions about how to best exploit the performance of modern storage technology leading to suboptimal designs, even if they were aware of the increasing improvements in storage technology. > In the process of writing this…

Yeah how many people are running apps on servers served at all or even partially by NVMe SSDs? Where I work for our on prem stuff it's basically all network storage.

There are a series of hardware updates and software bottlenecks to work through before access becomes more common, the performance will bubble up from below starting with more widespread NVMe devices, then faster NVMe over fabrics hardware will become more common, and drivers, hypervisors, filesystems, and storage apps will likely have to rethink things to re-optimize. That means different times in terms of showing up in the cloud/on-prem/etc.

Re: Modern storage is plenty fast, but the APIs are bad

#73

Post by _urga »

sgtnoodle wrote: post #39
_urga wrote: post #14

Earlier quoted context omitted.

The advertised bandwidth for RAM is not actually what you get per-core, which is what you care about in practice. If you want to know the upper bound on your per-core RAM bandwidth: 64 bytes (the size of a cache line) * 10 slots (in a CPU core's LFB or line fill buffer) / 100ns (the typical cost of a cache miss) * 1000000 * 1000 (to convert ns to ms to seconds) = 6400000000 bytes per second = 5.96 GiB per second RAM…

While I was in the hospital ICU earlier this year, I promised myself I would build a zen 3 desktop when it came out despite my 10 year old desktop still working just fine. I've since bought all the pieces but the CPU; they are all sold out. So I got a 6 core 3600XT in the interim. I bought fairly high binned RAM and overclocked it to 3600Mhz, and was surprised to cap out at about 36GB/s throughput. Your 6GiB/s per co…

Cool! I had a similar empirical experience working on a Cauchy Reed-Solomon encoder awhile back, which is essentially measuring xor speed, but I just couldn't get it past 6 GiB/s per core either, until I guessed I was hitting memory bandwidth limits. Only a few weeks ago I stumbled on the actual formula to work it out!

Re: Modern storage is plenty fast, but the APIs are bad

#74

Post by jeffbee »

masklinn wrote: post #60
jeffbee wrote: post #56

Earlier quoted context omitted.

I guess it depends on what you consider to be "typical". If you look at a many-core chip from AMD you'll see a gradient of access latency from one core to another. You'll see the same on any Intel Skylake-X descendant, although the slope of that gradient is less. Your software will need to be very highly optimized already before you start to sweat the difference, though.

Here's what jeffbee is talking about: https://www.anandtech.com/show/16214/amd-zen-3-ryzen-deep-di...

Yep. If you are sweating a microsecond, 100 nanoseconds is significant chunk of your budget. For this and possibly other reasons, a many-core CPU isn't always a great choice for hosted storage. If your goal is to export NVMe blocks over a network interface, you might be better off with an easier-to-program 4- or 8-core CPU. I don't like seeing 128 cores and a bunch of NVMe devices in the same box because it just causes trouble.

Re: Modern storage is plenty fast, but the APIs are bad

#75

Post by wtallis »

aidenn0 wrote: post #64
wtallis wrote: post #36

Earlier quoted context omitted.

> I expect the price of NVMe drives to drop over the next few years until they're cheap enough that the majority of computers are running NVMe drives. Price no longer has anything to do with it. PC OEMs are simply not shipping SATA SSDs any more, and major drive vendors have started to discontinue their client (OEM) SATA SSD product lines. We're just waiting for the SATA-based PC install base to be retired.

I have 8 SATA SSDs in my workstation; are there motherboards that could run a similar NVMe setup?

I wasn't including the workstation market when I referred to what PC OEMs are doing.

Are you using 8 consumer SATA SSDs in your workstation? Is it for the sake of increased capacity, or for the sake of increased performance? Because it's pretty easy now to match the performance of an 8-drive SATA RAID-0 with a single NVMe drive, but 8TB consumer NVMe SSDs are still 50% more expensive than 8TB consumer SATA SSDs.

(Also, even 8 SATA ports is above average for consumer motherboards; it looks like about 17% of the retail desktop motherboard models currently on the market have at least 8 SATA ports.)

Re: Modern storage is plenty fast, but the APIs are bad

#76

Post by kazinator »

This is a really poor article. Only in very rare circumstances can developers change the API's. API's are not "bad"; they are built to various important requirements. Only some of those requirements have to do with performance.

> “Well, it is fine to copy memory here and perform this expensive computation because it saves us one I/O operation, which is even more expensive”.

"I/O operation" in fact refers to the API call, not to the raw hardware operation. If the developer measured this and found it true, how can it be a misconception? It may be caused by a "bad" I/O API, but so what? The API is what it is.

API's provide one requirement which is stability: keeping applications working. That is king. You can't throw out API's every two years due to hardware advancements.

> “If we split this into multiple files it will be slow because it will generate random I/O patterns. We need to optimize this for sequential access and read from a single file”

Though solid state storage doesn't have track-to-track seek times, the sequential-access-fast rule of thumb has not become false.

Random access may have to wastefully read larger blocks of the data than are actually requested by the application. The unused data gets cached, but if it's not going to be accessed any time soon, it means that something else got wastefully bumped out of the cache. Sequential access is likely to make use of an entire block.

Secondly, there is that API again. The underlying operating system may provide a read-ahead mechanism which reduces its own overheads, benefiting the application which structures its data for sequential access, even if there is no inherent hardware-level benefit.

If there is any latency at all between the application and the hardware, and if you can guess what the application is going to read next, that's an opportunity to improve performance. You can correctly guess what the application will read if you guess that it is doing a sequential read, and the application makes that come true.

Re: Modern storage is plenty fast, but the APIs are bad

#77

Post by _urga »

wmf wrote: post #38
_urga wrote: post #14

Earlier quoted context omitted.

The advertised bandwidth for RAM is not actually what you get per-core, which is what you care about in practice. If you want to know the upper bound on your per-core RAM bandwidth: 64 bytes (the size of a cache line) * 10 slots (in a CPU core's LFB or line fill buffer) / 100ns (the typical cost of a cache miss) * 1000000 * 1000 (to convert ns to ms to seconds) = 6400000000 bytes per second = 5.96 GiB per second RAM…

What about prefetching? Tiger Lake gets over 20 GB/s per core. https://www.anandtech.com/show/16084/intel-tiger-lake-review...

Beats me!

Re: Modern storage is plenty fast, but the APIs are bad

#78

Post by Cojen »

I found this to be a good read, but I wish the author discussed the pros/cons of bypassing the file system and using a block device with direct I/O. I've found that with Optane drives the performance is high enough that the extra load from the file system (in terms of CPU) is significant. If the author was using a file system (which I assume is the case) which was it?

Re: Modern storage is plenty fast, but the APIs are bad

#79

Post by eightysixfour »

aidenn0 wrote: post #64
wtallis wrote: post #36

Earlier quoted context omitted.

> I expect the price of NVMe drives to drop over the next few years until they're cheap enough that the majority of computers are running NVMe drives. Price no longer has anything to do with it. PC OEMs are simply not shipping SATA SSDs any more, and major drive vendors have started to discontinue their client (OEM) SATA SSD product lines. We're just waiting for the SATA-based PC install base to be retired.

I have 8 SATA SSDs in my workstation; are there motherboards that could run a similar NVMe setup?

Yes, using PCIe expansion cards. I know of an AMD board that ships with 5 (3 on the board, 2 with a PCIe card). Could easily add more.

Re: Modern storage is plenty fast, but the APIs are bad

#80

Post by digikata »

aidenn0 wrote: post #64
wtallis wrote: post #36

Earlier quoted context omitted.

> I expect the price of NVMe drives to drop over the next few years until they're cheap enough that the majority of computers are running NVMe drives. Price no longer has anything to do with it. PC OEMs are simply not shipping SATA SSDs any more, and major drive vendors have started to discontinue their client (OEM) SATA SSD product lines. We're just waiting for the SATA-based PC install base to be retired.

I have 8 SATA SSDs in my workstation; are there motherboards that could run a similar NVMe setup?

Right now it's a bit more specialized to storage oriented server platforms that can run in the 10-40 NVMe devices. You get this sort of imbalance where any one or two high performance NVMe devices at full throughput can push more I/O than a single high end network link
Post reply on HN