Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

twitter.com

Post reply on HN
Print view View original

231–240 of 261 posts

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#231

Post by pajko »

Without any more information this post is just bullshit. For example, it's not documented how the flushing has been done. On Linux, even issuing 'sync' is not enough: https://unix.stackexchange.com/questions/98568/difference-be...

The bottom answer especially states that "blockdev --flushbufs may still be required if there is a large write cache and you're disconnecting the device immediately after"

The hpdarm utility has a parameter for syncing and flushing device buffers themselves. Seems like all three should be done for a complete flush at all levels.

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#232

Post by fulafel »

lxgr wrote: post #76
fulafel wrote: post #66

Earlier quoted context omitted.

I think this is typical behaviour with ext4 on Linux, if the application doesn't do fsync/fdatasync to flush the data to disk. Depending on mount options, ext4fs does metadata journaling ensuring the FS itself is not borked, but not data journaling which would safeguard the file contents in event of unclean shutdown with pending writes in the caches. The same phenomenon is at play when people complain that their log…

I don't think that's how it works: Flushing metadata before data would be a security concern (consider e.g. the metadata change of increasing a file's length due to an append before the data change itself), so file systems usually only ever do the opposite, which is safe. Getting back zeroes after a metadata sync (which must follow a data sync) would accordingly be an indication of something weird having happened at…

I think there could semi-reasonably be case for the zero bytes appearing if the fs knows there should be something written there, and the block is has been allocated, but no write yet. Then it's not compromising confidentiality to zero the allocated block when recovering the journal when the disk is mounted. But the zero byte origin doesn't seem to be spelled out anywhere so this is just off the cuff reasoning.

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#233

Post by nijave »

kmxdm wrote: post #217
nolist_policy wrote: post #213

Earlier quoted context omitted.

I still don't think this is the problem, the drive can just slow down accepting writes until it has reclaimed enough space. The bigger problem is manufacturers chasing the performance. Generally you get the feeling they just hit their firmware with a hammer so it barely doesn't break NTFS. See also the drama around btrfs' "unreliability", which is all traced back to drives with broken firmware. I fully expect bcachef…

Yeah, but then you have a write amplification problem. Padding is write amplification from the start, and then GC is invoked many more times than it otherwise would be. There is a fundamental problem with (truly) flushing an IO that is smaller than the media write unit. It will cause problems if "abused." The SSD either needs to take on the cost of mitigation (e.g. caps) or it needs some way to provide hints to the h…

I'm curious what the real cost of power protection is for the manufacturer. Adding caps or a bit of backup power seems like it _should_ be a fairly cheap compromise to maintain performance without lying about persistence

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#234

Post by yjftsjthsd-h »

vineyardmike wrote: post #129
yjftsjthsd-h wrote: post #52

Earlier quoted context omitted.

I was under the impression that a lot of off-brand USB devices didn't use the USB logo specifically to get around certification requirements. Basically, they just aren't advertising adherence to a standard. No idea about NVMe or BT.

I’ve similarly seen a lot of “TF card reader slots” instead of “micro SD card” slots.

Actually, I think that one has technical merit - the SD card spec includes non-storage IO - https://en.wikipedia.org/wiki/SD_card#SDIO_cards - that isn't supported by slots that only care about storage. At least, that's my understanding of the situation.

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#235

Post by matja »

worthless-trash wrote: post #194
matja wrote: post #172

Earlier quoted context omitted.

> which should prevent this problem if the hardware doesn't lie. Or, one can take the ZFS approach and assume the hardware often lies :)

I do not know how zfs will overcome hardware lying. If its going to fetch data that is in the drives cache, how will it overcome the persistence problem ?

Actual file data ends up in the same transaction group (txg) as metadata if both are changed within the same txg commit (either flushed explicitly, due to recordsize/buffer limit being reached, or txg commit timeout - 5 seconds by default), so if there is a write barrier violation caused by hardware lies followed by an untimely loss of power, the checksums for the txg updates won't match and they get rolled-back until the last valid one when importing the pool - which doesn't end up zero'ing out extents of a file (like in xfs) or ending up with a zero file size (like on ext3/ext4).

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#236

Post by nijave »

kristopolous wrote: post #154

Under long term heavy duty, I've routinely seen cheap modern platter outperform cheap brand name NVME. There's some cost cutting somewhere. The NVMEs can't seem to sustain throughput. It's been pretty disappointing to move I/O bound workloads over and not see notable improvements. The magnitude of data I'm talking about is 500-~3000GB I've only got two NVME machines for what I'm doing so I'll gladly accept that it's…

I used to use an HP EX920 for my system drive and it was abysmally slow at syncs. I'd open Signal and the computer would grind to a halt while it loaded messages from group chats. After much debugging, I found out Signal was saving each message to sqlite in a transaction causing lots of syncing.

I found some bash script that looped and wrote small blocks synchronously and the HP EX920 was like 20 syncs/sec and my WD RE4 spinner was around 150. Other SSDs were much faster (it was a few years ago so can't remember the exact numbers)

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#237

Post by nijave »

It'd be nice if there were a database of known bad/known good hardware to reference. I know there's been some spreadsheets and special purpose like the USB-C cables Benson Leung tested.

Especially for consumer hardware on Linux--there's a lot of stuff that "works" but is not necessarily stable long term or that required a lot of hacking on the kernel side to work around issues

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#238

Post by kmxdm »

nijave wrote: post #233
kmxdm wrote: post #217

Earlier quoted context omitted.

Yeah, but then you have a write amplification problem. Padding is write amplification from the start, and then GC is invoked many more times than it otherwise would be. There is a fundamental problem with (truly) flushing an IO that is smaller than the media write unit. It will cause problems if "abused." The SSD either needs to take on the cost of mitigation (e.g. caps) or it needs some way to provide hints to the h…

I'm curious what the real cost of power protection is for the manufacturer. Adding caps or a bit of backup power seems like it _should_ be a fairly cheap compromise to maintain performance without lying about persistence

M.2/2280 makes it hard. Can't use cheaper aluminum can capacitors due to their size/height. The low profile (tantalum?) capacitors are expensive and take up a lot of PCB area, forcing a two sided PCB design on 2280 (the 110mm version would be better here). M.2 only provides 5V. On other form factors you get 12V and can get more charge stored for the same capacitance (q=CV) without needing a DC-DC converter.

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#239

Post by supertrope »

tripdout wrote: post #60

Flushing in this case is from the SSDs internal DRAM cache to the actual NAND flash?

The DRAM cache does not hold user data. It holds the flash transition layer that links LBAs to NAND pages. Higher performance drives use 1GB of DRAM per 1TB of NAND. In cheap DRAM-less drives if the I/O to be serviced is not cached in the 1MB or so of SRAM it has to do a double lookup. Once to retrieve the full FTL table from NAND and a second lookup to actually service the I/O.

Re: I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)

#240

Post by fulafel »

danarmak wrote: post #196
worthless-trash wrote: post #194

Earlier quoted context omitted.

I do not know how zfs will overcome hardware lying. If its going to fetch data that is in the drives cache, how will it overcome the persistence problem ?

It will at the very least notice that the read data does not match the stored checksum and not return the garbage data to the application. In redundant (raidz) setups it will then read the data from another disk, and update the faulty disk. In a non-redundant setup (or if enough disks are corrupted) it will signal an IO error. An error is preferred to silently returning garbage data!

As an aside, can you still get the bad checksum file contents with zfs? Eg if it's a big database with its own checksums you might want to run a db level recovery on it.
Post reply on HN