Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unpowered SSDs slowly lose data

xda-developers.com

Post reply on HN
Print view View original

201–210 of 341 posts

Re: Unpowered SSDs slowly lose data

#201

Post by divan »

fairfeather wrote: post #54
dale_glass wrote: post #46

So on the off-chance that there's a firmware engineer in here, how does this actually work? Like does a SSD do some sort of refresh on power-on, or every N hours, or you have to access the specific block, or...? What if you interrupt the process, eg, having a NVMe in an external case that you just plug once a month for a few minutes to just use it as a huge flash drive, is that a problem? What about the unused space,…

SSD firmware engineer here. I work on enterprise stuff, so ymmv on consumer grade internals. Generally, the data refresh will all happen in the background when the system is powered (depending on the power state). Performance is probably throttled during those operations, so you just see a slightly slower copy while this is happening behind the scenes. The unused space decaying is probably not an issue, since the int…

I had to google what 'ymmv' means. To save other people's time – it's 'your mileage may vary'.

Re: Unpowered SSDs slowly lose data

#203

Post by 2WSSd-JzVM »

breput wrote: post #157

The spinrite[0] user group has noticed some of these effects, even on in-service drives. The theory is that operating system files, which rarely change, are written and almost never re-written. So the charges begin to decay over time and while they might not be unreadable, reads for these blocks require additional error correction, which reduces performance. There have been a significant number of (anecdotal) reports…

If full rewrite helps doesn't it sound like TRIM implementation in SSDs is buggy or insufficient? Or internal cell wear-maps aren't detailed enough. Anyway plenty ways it can go wrong, SSD firmware had also plenty of high profile bugs, including total bricking.

Re: Unpowered SSDs slowly lose data

#204

Post by antonkochubey »

adrian_b wrote: post #196
londons_explore wrote: post #155

Earlier quoted context omitted.

I'm sad that drives don't have a 'shutdown' command which writes a few extra bytes of ECC data per page into otherwise empty flash cells. It turns out that a few extra bytes can turn a 1 year endurance into a 100 year endurance.

There are programs with which you can add any desired amount of redundancy to your backup archives, so that they would survive corruption that does not affect a greater amount of data than the added redundancy. For instance, on Linux there is par2cmdline. For all my backups, I create pax archives, which are then compressed, then encrypted, then expanded with par2create, then aggregated again in a single pax file (the…

Unfortunately some SSD controllers plainly refuse to read data they consider corrupted, even if you have extra parity that could potentially restore corrupted data, your entire drive might refuse to read.

Re: Unpowered SSDs slowly lose data

#205

Post by mbreese »

dotancohen wrote: post #185
mbreese wrote: post #107

Earlier quoted context omitted.

Honestly this is one of my favorite things about ZFS. I know that a disk scan is performed every week/month (whatever schedule). And I also know that it has verified the contents of each block. It is very reassuring in that way.

In threads like this I keep hearing about ZFS. What would be the drawbacks of running ZFS as a home user? I keep my OS on the SSD and my files on spinning rust, if that's relevant.

1) you have to have an OS that supports it.

2) even if your OS supports it, you may have difficulty using it for your root volume, so partitioning is probably required.

2a) in your case you may not want to use it on your boot volume which would negate the SSD benefit for you.

3) it is recommended that you have ECC RAM due to the checksums. This isn’t a hard and fast requirement, but it does make you more resilient to bitflips.

4) it isn’t the absolute fastest file system. But it’s not super slow. There are caching options for read and write that benefit from SSDs, but you’re just adding costs here to get speed increases.

I only use it on servers or NASs. The extra hassles of using it on a workstation keep me from running it on a laptop. Unless you want to use FreeBSD that is… then you’d be fine (and FreeBSD is pretty usable as a daily driver). Realistically, I’m not sure how practical it is for most home users. But it is an example of what a filesystem can offer when it is well designed.

Re: Unpowered SSDs slowly lose data

#206

Post by lxgr »

adrian_b wrote: post #198
lxgr wrote: post #193

Earlier quoted context omitted.

Wrong layer. SSDs know which blocks have been written to a lot, have been giving a lot of read errors before etc., and often even have heterogeneous storages (such as a bit of SLC for burst writing next to a bunch of MLC for density). They can spend ECC bits much more efficiently with that information than a file system ever could, which usually sees the storage as a flat, linear array of blocks.

This is true, but nevertheless you cannot place your trust only in the manufacturer of the SSD/HDD, as I have seen enough cases when the SSD/HDD reports no errors, but nonetheless it returns corrupted data. For any important data you should have your own file hashes, for corruption detection, and you should add some form of redundancy for file repair, either with a specialized tool or simply by duplicating the file o…

Verifying at higher layers can be ok (it's still not ideal!), but trying to actively fix things below that are broken usually quickly becomes a nightmare.

Re: Unpowered SSDs slowly lose data

#207

Post by anthk »

ACCount37 wrote: post #162
userbinator wrote: post #77

One key point about retention which is not often mentioned, and indeed neither does this article, is that retention is inversely proportional to program/erase cycles and decreases exponentially with increasing temperature. Hence why retention specs are usually X amount of time after Y cycles at Z temperature. Even a QLC SSD that has only been written to once, and kept in a freezer at -40, may hold data for several de…

Because no one is willing to pay for SLC. Those QLC NAND chips? Pretty much all of them have an "SLC mode", which treats each cell as 1 bit, and increases both write speeds and reliability massively. But who wants to have 4 times less capacity for the same price?

Myself wants. I remember when the UBIFS module (or some kernel settings) for the Debian kernel was MLC against SLC. You could store 4X more data now, but at a cost of really bad reability: A SINGLE bad shutdown and your partitions would be corrupted up to the point of not being able to properly boot any more, having to reflash the NAND.

Re: Unpowered SSDs slowly lose data

#209

Post by DeepSeaTortoise »

lxgr wrote: post #193
consp wrote: post #166

Earlier quoted context omitted.

Blind question with no attempt to look it up: why don't filesystems do this? It won't work for most boot code but that is relatively easy to fix by plugging it in somewhere else.

Wrong layer. SSDs know which blocks have been written to a lot, have been giving a lot of read errors before etc., and often even have heterogeneous storages (such as a bit of SLC for burst writing next to a bunch of MLC for density). They can spend ECC bits much more efficiently with that information than a file system ever could, which usually sees the storage as a flat, linear array of blocks.

IMO it's exactly the right layer, just like for ECC memory.

There's a lot of potential for errors when the storage controller processes and turns the data into analog magic to transmit it.

In practice, this is a solved problem, but only until someone makes a mistake, then there will be a lot of trouble debugging it between the manufacturer certainly denying their mistake and people getting caught up on the usual suspects.

Doing all the ECC stuff right on the CPU gives you all the benefits against bitrot and resilience against all errors in transmission for free.

And if all things go just right we might even be getting better instruction support for ECC stuff. That'd be a nice bonus

Re: Unpowered SSDs slowly lose data

#210

Post by throwaway290 »

userbinator wrote: post #168
ACCount37 wrote: post #162

Earlier quoted context omitted.

Because no one is willing to pay for SLC. Those QLC NAND chips? Pretty much all of them have an "SLC mode", which treats each cell as 1 bit, and increases both write speeds and reliability massively. But who wants to have 4 times less capacity for the same price?

4 times less capacity but 100x or more endurance or retention at the same price looks like a great deal to me. Alternatively: do you want to have 4x more capacity at 1/100th the reliability? Plenty of people would be willing to pay for SLC mode. There is an unofficial firmware hack that enables it: https://news.ycombinator.com/item?id=40405578 1TB QLC SSDs are But why won't the manufacturers let you choose? The real…

> I have an old SLC USB drive which is only 512MB, but it's nearly 20 years old and some of the very first files I wrote to it are still intact (I last checked several months ago

It's not about age of drive. It's how much time it spent without power.

Post reply on HN