Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

twitter.com

Post reply on HN
Print view View original

171–180 of 261 posts

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#171

Post by chrismorgan »

r1ch wrote: post #54

We shipped a shader cache in the latest release of OBS and quickly had reports come in that the cached data was invalid. After investigating, the cache files were the correct size on disk but the contents were all zero. On a journaled file system this seems like it should be impossible, so the current guess is that some users have SSDs that are ignoring flushes and experience data corruption on crash / power loss.

I lost Audacity projects due to BSODs on a Surface Book several times in ~2019: the *_data/**.au files were intact, each containing just a few seconds of audio; but the .aup XML file that maps them and contains whatever else makes up the project was all zeroed. My memory’s fuzzy, but I think it was something like exit sometimes triggering the BSOD, and save-on-exit corrupting consistently if it BSODed, and so the workaround was to remember to save first, and then if it BSODs you’re OK.

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#172

Post by matja »

Dylan16807 wrote: post #78
fulafel wrote: post #66

Earlier quoted context omitted.

I think this is typical behaviour with ext4 on Linux, if the application doesn't do fsync/fdatasync to flush the data to disk. Depending on mount options, ext4fs does metadata journaling ensuring the FS itself is not borked, but not data journaling which would safeguard the file contents in event of unclean shutdown with pending writes in the caches. The same phenomenon is at play when people complain that their log…

The current default is data=ordered, which should prevent this problem if the hardware doesn't lie. The data doesn't go in the journal, but it has to be written before the journal is committed. There was a point where ext3 defaulted to data=writeback, which can definitely give you files full of null bytes. And data=journal exists but is overkill for this situation.

> which should prevent this problem if the hardware doesn't lie.

Or, one can take the ZFS approach and assume the hardware often lies :)

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#173

Post by vladvasiliu »

IngvarLynn wrote: post #40

There is a flood of fake SSDs currently, mostly big brands. I've recently purchased counterfeit 1TB. It passes all the tests, performance is ok, it works... except it gets episodes where ioping would be anything between 0.7 ms and 15 seconds, that is under zero load. And these are quality fakes from a physical appearance perspective. The only way I could tell mine was fake is that the official Kingston firmware updat…

That's interesting. I have a Samsung 990 pro bought on Amazon and have the random lags. I've only noticed it in the terminal, so I figured something else may be the culprit. Never went to 15 secondes, but it can be around 1s.

The Samsung Magician app on Windows reports it as "genuine" and it was able to apply two firmware updates. The only thing it complains about is that I should be using PCIE 4 instead of 3, but I can't do anything about that.

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#174

Post by dspillett »

Sakos wrote: post #161
wannacboatmovie wrote: post #67

Earlier quoted context omitted.

They may be pointing to unallocated space which on a SSD running TRIM would return all zeros. NTFS is an extremely resilient yet boring filesystem, I cannot remember the last time I had to run chkdsk even after an improper shutdown.

As somebody who worked as a PC technician for a while until very recently, I've run chkdsk and had to repair errors on NTFS filesystems very, very, very often. It's almost an everyday thing. Anecdotal evidence is less than useful here.

So anecdotal evidence is not useful, as proven by your anecdotal evidence? :)

FWIW I've found NTFS and ext3/4 to be of similar reliability over the years, in general use and in the face of improper shutdown. Metadata journaling does a lot to preserve the filesystem in such circumstances. Most of the few significant problems I've had have been due to hardware issues, which few filesystems on their own will help you with.

It is worth noting that when you run tools like chkdsk or fsck, some of the issues reported and fixed are not data damaging, or structurally dangerous, or at least not immediately so. For instance free areas marked in such a way that makes them look used to the allocation algorithms.

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#175

Post by vladvasiliu »

kristopolous wrote: post #170
vladvasiliu wrote: post #168

Earlier quoted context omitted.

I don't know about the 950 pro specifically, but when I bought my 980 pro I looked into this, and it seemed that this drive does have a drop in write speed after a while (can't remember how long) but "low speed" wasn't that low. Again, don't remember specifics, but it was above 1 GB/s. Other drives fared much worse, with a drop coming in sooner and going lower. Depending on your needs this can be an issue. For me, us…

Thanks. I guess I can't really blame them for cutting corners on a use case that 99.9% of their customers never see. I'd probably advocate the same.

I think it's important to be able to have a rough idea of how you'd be using this. If this drop in performance means I could use a cheaper drive, I'm all for it.

But some drives are truly awful. My work laptop came with a cheap Samsung drive that would quickly drop to around 200 MB/s. At first, I thought I had somehow badly configured Linux or something (I'm running zfs with native encryption).

Then I went and checked my desktop running quite worn-out SATA ssds from ~2012 (840 evo) and those drives would wipe the floor with the NVMe in write performance. They wouldn't go below 400 something MB/s until almost full. Same kernel version and zfs config.

It would seem that this is quite common behavior in cheap drives. But I guess that if all you do is browse the web, write mails in outlook and type the occasional word document, you're still ahead of spinning rust for the durability (it won't break if you drop your laptop) and for the latency.

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#176

Post by agbrrw »

r1ch wrote: post #54

We shipped a shader cache in the latest release of OBS and quickly had reports come in that the cached data was invalid. After investigating, the cache files were the correct size on disk but the contents were all zero. On a journaled file system this seems like it should be impossible, so the current guess is that some users have SSDs that are ignoring flushes and experience data corruption on crash / power loss.

>experience data corruption on crash / power loss

You mean on complete system crash, right? Your application crashing shouldn't lead to files being fulls of zeroes as long as you've already written everything out.

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#177

Post by slowmotiony »

iamacyborg wrote: post #140
alanfranz wrote: post #59

Earlier quoted context omitted.

Did you get the fake in an official box? Or OEM version? This is quite a big claim.

It doesn't strike me as being a big claim, I recently bought some RAM for a NUC a few weeks ago on Amazon only to determine that it was likely counterfeit. It came in an official box with all packaging intact.

Then how did you determine it was fake?

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#178

Post by kristopolous »

vladvasiliu wrote: post #175
kristopolous wrote: post #170

Earlier quoted context omitted.

Thanks. I guess I can't really blame them for cutting corners on a use case that 99.9% of their customers never see. I'd probably advocate the same.

I think it's important to be able to have a rough idea of how you'd be using this. If this drop in performance means I could use a cheaper drive, I'm all for it. But some drives are truly awful. My work laptop came with a cheap Samsung drive that would quickly drop to around 200 MB/s. At first, I thought I had somehow badly configured Linux or something (I'm running zfs with native encryption). Then I went and checke…

In my use case I'm talking about arrays of them, think 8. RAID can parallelize platter really nicely in that configuration. But on modern ram/cpu (epyc 9654s), you'll still see the disk dragging you down. NVME drags me down more.

Maybe the key is a bunch of small ones. Like 20 512GB modules... That may be brilliant. It's way cheap

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#179

Post by vladvasiliu »

kristopolous wrote: post #178
vladvasiliu wrote: post #175

Earlier quoted context omitted.

I think it's important to be able to have a rough idea of how you'd be using this. If this drop in performance means I could use a cheaper drive, I'm all for it. But some drives are truly awful. My work laptop came with a cheap Samsung drive that would quickly drop to around 200 MB/s. At first, I thought I had somehow badly configured Linux or something (I'm running zfs with native encryption). Then I went and checke…

In my use case I'm talking about arrays of them, think 8. RAID can parallelize platter really nicely in that configuration. But on modern ram/cpu (epyc 9654s), you'll still see the disk dragging you down. NVME drags me down more. Maybe the key is a bunch of small ones. Like 20 512GB modules... That may be brilliant. It's way cheap

That sounds like you might be having quite specific needs. Some reviews have graphs showing how the drives behave in various situations, so that could help you.

However, what I've found perusing those reviews is that there's a huge price gap between this class of drives (9x0 pro, wd 8xx and the like) and "enterprise" drives which seem to have more stable performance.

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#180

Post by the8472 »

Dylan16807 wrote: post #109
altfredd wrote: post #103

Earlier quoted context omitted.

The "data" setting of ext filesystems isn't replacement for fsync().

It's not a replacement but it can give you some guarantees. Also fsync is a terrible API that should be replaced, but that's mostly a different topic.

At least on linux you can use io_uring to make fsync asynchronous. And you can initiate some preparatory flushing with sync_file_range and only do the final commit with fsync to cut down the latency.
Post reply on HN