> unpowered SSDs slowly lose data so it's as if the data... rusts , a little bit at a time
Usually described as bit rot.
Unpowered SSDs slowly lose data
211–220 of 341 posts
Re: Unpowered SSDs slowly lose data
#212[dead]
Re: Unpowered SSDs slowly lose data
#213Earlier quoted context omitted.
There are programs with which you can add any desired amount of redundancy to your backup archives, so that they would survive corruption that does not affect a greater amount of data than the added redundancy. For instance, on Linux there is par2cmdline. For all my backups, I create pax archives, which are then compressed, then encrypted, then expanded with par2create, then aggregated again in a single pax file (the…
Unfortunately some SSD controllers plainly refuse to read data they consider corrupted, even if you have extra parity that could potentially restore corrupted data, your entire drive might refuse to read.
Huh?
The issue being discussed is random blocks, yes?
If your entire drive is bricked, that is an entirely different issue.
Re: Unpowered SSDs slowly lose data
#214Earlier quoted context omitted.
Wrong layer. SSDs know which blocks have been written to a lot, have been giving a lot of read errors before etc., and often even have heterogeneous storages (such as a bit of SLC for burst writing next to a bunch of MLC for density). They can spend ECC bits much more efficiently with that information than a file system ever could, which usually sees the storage as a flat, linear array of blocks.
IMO it's exactly the right layer, just like for ECC memory. There's a lot of potential for errors when the storage controller processes and turns the data into analog magic to transmit it. In practice, this is a solved problem, but only until someone makes a mistake, then there will be a lot of trouble debugging it between the manufacturer certainly denying their mistake and people getting caught up on the usual susp…
> There's a lot of potential for errors when the storage controller processes and turns the data into analog magic to transmit it.
That's a physical layer, and as such should obviously have end-to-end ECC appropriate to the task. But the error distribution shape is probably very different from that of bytes in NAND data at rest, which is different from that of DRAM and PCI again.
For the same reason, IP does not do error correction, but rather relies on lower layers to present error-free datagram semantics to it: Ethernet, Wi-Fi, and (managed-spectrum) 5G all have dramatically different properties that higher layers have no business worrying about. And sticking with that example, once it becomes TCP's job to handle packet loss due to transmission errors (instead of just congestion), things go south pretty quickly.
Re: Unpowered SSDs slowly lose data
#215Earlier quoted context omitted.
Blind question with no attempt to look it up: why don't filesystems do this? It won't work for most boot code but that is relatively easy to fix by plugging it in somewhere else.
The filesystem doesn't have access to the right existing ECC data to be able to add a few bytes to do the job. It would need to store a whole extra copy. There are potentially ways a filesystem could use heirarchical ECC to just store a small percentage extra, but it would be far from theoretically optimal and rely on the fact just a few logical blocks of the drive become unreadable, and those logical blocks aren't c…
Reed Solomon codes, or forward error correction is what you’re discussing. All modern drives do it at low levels anyway.
It would not be hard for a COW file system to use them, but it can easily get out of control paranoia wise. Ideally you’d need them for every bit of data, including metadata.
That said, I did have a computer that randomly bit flipped when writing to storage sometimes (eventually traced it to an iffy power supply), and PAR (a type of reed solomon coding forward error correction library) worked great for getting a working backup off the machine. Every other thing I tried would end up with at least a couple bit flip errors per GB, which make it impossible.
Re: Unpowered SSDs slowly lose data
#216One key point about retention which is not often mentioned, and indeed neither does this article, is that retention is inversely proportional to program/erase cycles and decreases exponentially with increasing temperature. Hence why retention specs are usually X amount of time after Y cycles at Z temperature. Even a QLC SSD that has only been written to once, and kept in a freezer at -40, may hold data for several de…
I'm sad that drives don't have a 'shutdown' command which writes a few extra bytes of ECC data per page into otherwise empty flash cells. It turns out that a few extra bytes can turn a 1 year endurance into a 100 year endurance.
I guess the only way to do this today is with a raid array?
Re: Unpowered SSDs slowly lose data
#217Earlier quoted context omitted.
noob question... how do i force a full read?
On macOS / Linux you can use `dd` to "copy" everything from /dev/yourssd to /dev/null. Just be careful not to do it the other way! https://www.man7.org/linux/man-pages/man1/dd.1.html I have no idea if forcing a read is good / the right way. I'm just answering how to do it.
/dev/null is "empty" if you read it, its /dev/zero you dont want to dd onto your disk.
Re: Unpowered SSDs slowly lose data
#218Earlier quoted context omitted.
Unfortunately some SSD controllers plainly refuse to read data they consider corrupted, even if you have extra parity that could potentially restore corrupted data, your entire drive might refuse to read.
Huh? The issue being discussed is random blocks, yes? If your entire drive is bricked, that is an entirely different issue.
Here’s the thing. That SSD controller is the interface between you and those blocks.
If it decides, by some arbitrary measurement, as defined by some logic within its black box firmware, that it should stop returning all blocks, then it will do so, and you have almost no recourse.
This is a very common failure mode of SSDs. As a consequence of some failed blocks (likely exceeding a number of failed blocks, or perhaps the controller’s own storage failed), drives will commonly brick themselves.
Perhaps you haven’t seen it happen, or your SSD doesn’t do this, or perhaps certain models or firmwares don’t, but some certainly do, both from my own experience, and countless accounts I’ve read elsewhere, so this is more common than you might realise.
Re: Unpowered SSDs slowly lose data
#219Does the same apply to USB thumb drives, i.e. do they lose their data if not plugged in?
USB thumb drives (and SD cards) are bottom-of-the-barrel flash, so yes, absolutely.
Re: Unpowered SSDs slowly lose data
#220Earlier quoted context omitted.
More certain to just do a full read of the drive to force error correction and updating of any weakening data.
noob question... how do i force a full read?
This is the most straightforward reliable option:
> sudo dd if=/dev/sdX of=/dev/null bs=4M status=progress iflag=direct