Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

twitter.com

Post reply on HN
Print view View original

211–220 of 261 posts

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#211

Post by vladvasiliu »

thargor90 wrote: post #199
vladvasiliu wrote: post #173

Earlier quoted context omitted.

That's interesting. I have a Samsung 990 pro bought on Amazon and have the random lags. I've only noticed it in the terminal, so I figured something else may be the culprit. Never went to 15 secondes, but it can be around 1s. The Samsung Magician app on Windows reports it as "genuine" and it was able to apply two firmware updates. The only thing it complains about is that I should be using PCIE 4 instead of 3, but I…

I have been able to fix these random lags by doing multiple full disk reads. The first one will take very long, because it will trigger these lags. Subsequent ones will be much better. The leading theory I have read is that maintenance/refreshing on the ssd is not done preventative/correctly by the firmware and you need to trigger it by accessing the data.

I'm going to try that, but I have little hope since this happened ever since the drive was brand new.

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#212

Post by vladvasiliu »

Sakos wrote: post #190
kristopolous wrote: post #178

Earlier quoted context omitted.

In my use case I'm talking about arrays of them, think 8. RAID can parallelize platter really nicely in that configuration. But on modern ram/cpu (epyc 9654s), you'll still see the disk dragging you down. NVME drags me down more. Maybe the key is a bunch of small ones. Like 20 512GB modules... That may be brilliant. It's way cheap

For an example of an SSD review that shows a sustained write test: https://www.tomshardware.com/reviews/samsung-980-m2-nvme-ssd... ( https://cdn.mos.cms.futurecdn.net/mzdXcBUJUuxkbYfmQigqQE-970... ) In a 900 second timeframe, the 980 Pro drops down to a bit above 1GB/s, while the worst is somewhere in the tens of MB/s. So there are significant differences in SSD performance profiles which you can only find out about…

This was the kind of review I was talking about. But you should also look at the specific capacity you're after. IIRC for the 980 non pro the 1 TB is better than the smaller ones.

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#213

Post by nolist_policy »

kmxdm wrote: post #206
gumby wrote: post #197

Earlier quoted context omitted.

The whole point of explicit flush is to tell the drive that you want the write at the expense of performance. Either the drive should not accept the flush command or it should fulfill it, not lie. (BTW this points out the crappy use of the word “performance” in computing to mean nothing but “speed”. The machine should “perform” what the user requests — if you hired someone to do a task and they didn’t do it, we’d say…

The more dire problem is the case where the drive runs out of physical capacity before logical capacity. If the host flushes data that is smaller than the physical write unit of the SSD, capacity is lost to padding (if the SSD honors every Flush). A "reasonable" amount of Flush would not make too much of a difference, but a pathological case like flush-after-every-4k would cause the SSD to run out of space prematurel…

I still don't think this is the problem, the drive can just slow down accepting writes until it has reclaimed enough space.

The bigger problem is manufacturers chasing the performance. Generally you get the feeling they just hit their firmware with a hammer so it barely doesn't break NTFS.

See also the drama around btrfs' "unreliability", which is all traced back to drives with broken firmware. I fully expect bcachefs will get exactly the same problems.

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#214

Post by Hizonner »

nolist_policy wrote: post #213
kmxdm wrote: post #206

Earlier quoted context omitted.

The more dire problem is the case where the drive runs out of physical capacity before logical capacity. If the host flushes data that is smaller than the physical write unit of the SSD, capacity is lost to padding (if the SSD honors every Flush). A "reasonable" amount of Flush would not make too much of a difference, but a pathological case like flush-after-every-4k would cause the SSD to run out of space prematurel…

I still don't think this is the problem, the drive can just slow down accepting writes until it has reclaimed enough space. The bigger problem is manufacturers chasing the performance. Generally you get the feeling they just hit their firmware with a hammer so it barely doesn't break NTFS. See also the drama around btrfs' "unreliability", which is all traced back to drives with broken firmware. I fully expect bcachef…

> so it barely doesn't break NTFS.

That's very optimistic of you.

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#215

Post by gsich »

WendyTheWillow wrote: post #151
andybak wrote: post #149

Earlier quoted context omitted.

"users of my application". You seem to be claiming that everyone still on Twitter is ideologically compromised. There's a ton of people just ignoring the politics and I still need a channel to reach them.

The politics are all that are left; it's impossible to use Twitter and not participate on some level. For example, Twitter is now called X. Surely they noticed that.

>For example, Twitter is now called X. Surely they noticed that.

So what?

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#216

Post by dtx1 »

I am a bit annoyed that everyone here takes this at face value. There's 0 evidence given, not even the vendors and models are named to confirm this.

On a related note I tested 4 DDR5 Ram kits from major vendors - half of them corrupt data when exposed to UV light.

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#217

Post by kmxdm »

nolist_policy wrote: post #213
kmxdm wrote: post #206

Earlier quoted context omitted.

The more dire problem is the case where the drive runs out of physical capacity before logical capacity. If the host flushes data that is smaller than the physical write unit of the SSD, capacity is lost to padding (if the SSD honors every Flush). A "reasonable" amount of Flush would not make too much of a difference, but a pathological case like flush-after-every-4k would cause the SSD to run out of space prematurel…

I still don't think this is the problem, the drive can just slow down accepting writes until it has reclaimed enough space. The bigger problem is manufacturers chasing the performance. Generally you get the feeling they just hit their firmware with a hammer so it barely doesn't break NTFS. See also the drama around btrfs' "unreliability", which is all traced back to drives with broken firmware. I fully expect bcachef…

Yeah, but then you have a write amplification problem. Padding is write amplification from the start, and then GC is invoked many more times than it otherwise would be. There is a fundamental problem with (truly) flushing an IO that is smaller than the media write unit. It will cause problems if "abused." The SSD either needs to take on the cost of mitigation (e.g. caps) or it needs some way to provide hints to the host that don't exist today.

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#219

Post by HankB99 »

jtwaleson wrote: post #147
cm2187 wrote: post #134

Earlier quoted context omitted.

To me the real thing missing is whether those drive advertise power loss protection or not. The next question is whether they are to be used in a laptop where power loss protection is less relevant given the local battery.

That should be irrelevant, because flush is flush right? If your SSD does not write the data after a flush it's violating basic hard drive functionality.

GP has a point in that a laptop is much less likely to experience unplanned power loss because of the built in battery. However that does not help desktops which also use NVME SSDs.

The other problem is that this doesn't only cause problems resulting from power loss. At least some files systems guarantee consistency of data on the drive by flushing at critical points. ZFS on Linux does this. If the flush doesn't happen as promised, subsequent writes could result in corrupted files should something else like a crash interrupt operation.

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#220

Post by colanderman »

lxgr wrote: post #76
fulafel wrote: post #66

Earlier quoted context omitted.

I think this is typical behaviour with ext4 on Linux, if the application doesn't do fsync/fdatasync to flush the data to disk. Depending on mount options, ext4fs does metadata journaling ensuring the FS itself is not borked, but not data journaling which would safeguard the file contents in event of unclean shutdown with pending writes in the caches. The same phenomenon is at play when people complain that their log…

I don't think that's how it works: Flushing metadata before data would be a security concern (consider e.g. the metadata change of increasing a file's length due to an append before the data change itself), so file systems usually only ever do the opposite, which is safe. Getting back zeroes after a metadata sync (which must follow a data sync) would accordingly be an indication of something weird having happened at…

The file's size could have been set by the application before copying data to it. This will result in a file which reads all zeroes.

Or if it were a hardware ordering fault, remember that SSD TRIM is typically used by modern filesystems to reclaim unused space. TRIMmed blocks read as zero.

Post reply on HN