Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

twitter.com

Post reply on HN
Print view View original

201–210 of 261 posts

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#201

Post by donmcronald »

kmxdm wrote: post #195

Writes are completed to the host when they land on the SSD controller, not when written to Flash. The SSD controller has to accumulate enough data to fill its write unit to Flash (the absolute minimum would be a Flash page, typically 16kB). If it waited for the write to Flash to send a completion, the latency would be unbearable. If it wrote every write to Flash as quickly as possible, it could waste much of the driv…

> Non-consumer drives solve the problem with back-up capacitance.

I’m pretty sure they used to be on consumer drives too. Then they got removed and all the review sites gave the manufacturer a free pass even though they’re selling products that are inadequate.

Disks have one job, save data. If they can’t do that reliably they’re defective IMO.

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#203

Post by nippoo »

kmxdm wrote: post #195

Writes are completed to the host when they land on the SSD controller, not when written to Flash. The SSD controller has to accumulate enough data to fill its write unit to Flash (the absolute minimum would be a Flash page, typically 16kB). If it waited for the write to Flash to send a completion, the latency would be unbearable. If it wrote every write to Flash as quickly as possible, it could waste much of the driv…

This is the whole point of a FLUSH though. You expect latency penalties and worse performance (and extra pages) if you flush, but that's the expected behaviour: not for it to (apparently) completely disregard the command while pretending like it's done it.

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#204

Post by catdog »

dspillett wrote: post #183
wmf wrote: post #87

Earlier quoted context omitted.

SK Hynix is a major brand and the P31 is a great midrange SSD... except for the fact that it seemingly doesn't care about your data.

> SK Hynix is a major brand Is it? I passed on an offer for a drive carrying that name, and got something else for slightly more, the other day as I didn't know the name. Perhaps their noteworthiness varies internationally? Or do they mainly sell to manufacturers rather than direct to the likes of me?

They are even among the manufacturers of actual flash memory chips.

> Or do they mainly sell to manufacturers rather than direct to the likes of me?

This. Think they mostly sell OEM SSDs under their name, if you buy a laptop or pre built system from a major manufacturer chances are not that low that you find a SK Hynix SSD in there.

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#205

Post by daneel_w »

ricardobeat wrote: post #44

Misleading headline since after testing eight more drives, none more failed. 2/12 is not nearly as dramatic as “half”, and the ones that lost data are the cheap brands as one would expect.

> "... and the ones that lost data are the cheap brands as one would expect."

What a sad world to live in, when one comes to expect cheap storage devices to not fulfill intended function.

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#206

Post by kmxdm »

gumby wrote: post #197
kmxdm wrote: post #195

Writes are completed to the host when they land on the SSD controller, not when written to Flash. The SSD controller has to accumulate enough data to fill its write unit to Flash (the absolute minimum would be a Flash page, typically 16kB). If it waited for the write to Flash to send a completion, the latency would be unbearable. If it wrote every write to Flash as quickly as possible, it could waste much of the driv…

The whole point of explicit flush is to tell the drive that you want the write at the expense of performance. Either the drive should not accept the flush command or it should fulfill it, not lie. (BTW this points out the crappy use of the word “performance” in computing to mean nothing but “speed”. The machine should “perform” what the user requests — if you hired someone to do a task and they didn’t do it, we’d say…

The more dire problem is the case where the drive runs out of physical capacity before logical capacity. If the host flushes data that is smaller than the physical write unit of the SSD, capacity is lost to padding (if the SSD honors every Flush). A "reasonable" amount of Flush would not make too much of a difference, but a pathological case like flush-after-every-4k would cause the SSD to run out of space prematurely. There should be a better interface to handle all this, but the IO stack would need to be modified to solve what amounts to a cost issue at the SSD level. It's a race to the bottom selling 1TB consumer SSDs for less than $100.

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#207

Post by kaliszad »

kristopolous wrote: post #178
vladvasiliu wrote: post #175

Earlier quoted context omitted.

I think it's important to be able to have a rough idea of how you'd be using this. If this drop in performance means I could use a cheaper drive, I'm all for it. But some drives are truly awful. My work laptop came with a cheap Samsung drive that would quickly drop to around 200 MB/s. At first, I thought I had somehow badly configured Linux or something (I'm running zfs with native encryption). Then I went and checke…

In my use case I'm talking about arrays of them, think 8. RAID can parallelize platter really nicely in that configuration. But on modern ram/cpu (epyc 9654s), you'll still see the disk dragging you down. NVME drags me down more. Maybe the key is a bunch of small ones. Like 20 512GB modules... That may be brilliant. It's way cheap

Or just get some Kioxia CD8 or CD8R or similar from Samsung/ Solidigm/ Micron depending on what you need. It will be much faster than spinning rust in all situations I am quite sure. Decent SSDs have a huge latency advantage compared to even the best HDDs just out of principle. Enterprise SSDs focused on read-write workloads can sustain decent performance even under continuous 100% load. For example: https://apac.kioxia.com/en-apac/business/ssd/data-center-ssd...

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#208

Post by nolist_policy »

danarmak wrote: post #196
worthless-trash wrote: post #194

Earlier quoted context omitted.

I do not know how zfs will overcome hardware lying. If its going to fetch data that is in the drives cache, how will it overcome the persistence problem ?

It will at the very least notice that the read data does not match the stored checksum and not return the garbage data to the application. In redundant (raidz) setups it will then read the data from another disk, and update the faulty disk. In a non-redundant setup (or if enough disks are corrupted) it will signal an IO error. An error is preferred to silently returning garbage data!

The "zeroed-out file" problem is not about firmware lying though, it is about applications using fsync() wrongly or not at all. Look up the O_PONIES controversy.

Sure, due to their COW nature zfs and btrfs provide better behavior despite broken applications. But you can't solve persistence in the face of lying firmware.

Even thought zfs has some enhancements to not corrupt itself on such drives, if you run for example a database on top, all guarantees around commit go out the window.

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#210

Post by kmxdm »

nolist_policy wrote: post #198
kmxdm wrote: post #195

Writes are completed to the host when they land on the SSD controller, not when written to Flash. The SSD controller has to accumulate enough data to fill its write unit to Flash (the absolute minimum would be a Flash page, typically 16kB). If it waited for the write to Flash to send a completion, the latency would be unbearable. If it wrote every write to Flash as quickly as possible, it could waste much of the driv…

> If a host tried to flush after every write to force the latter behavior, it would end up with the same problem. So? No reason to break the contract that flush makes all submitted writes durable. The drive can compact space in the background.

Yes, GC should be smart enough to free up space from padding. But then there's a write amplification penalty and meeting endurance specifications is impossible. A padded write already carries a write amplification >1, then GC needs to be invoked much more frequently on top of that to drive it even higher. With pathological Flush usage, you have to pick your poison. Run out of space, run out of SSD life.
Post reply on HN