Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unpowered SSDs slowly lose data

xda-developers.com

Post reply on HN
Print view View original

311–320 of 341 posts

Re: Unpowered SSDs slowly lose data

#311

Post by adrian_b »

casenmgreen wrote: post #293
adrian_b wrote: post #196

Earlier quoted context omitted.

There are programs with which you can add any desired amount of redundancy to your backup archives, so that they would survive corruption that does not affect a greater amount of data than the added redundancy. For instance, on Linux there is par2cmdline. For all my backups, I create pax archives, which are then compressed, then encrypted, then expanded with par2create, then aggregated again in a single pax file (the…

Thank you for this. I had no knowledge of pax, or that par was an open standard, and I care about what they help with. Going to switch over to using both in my backups.

For handling pax archives, I recommend the "libarchive" package, which is available in many Linux distributions, even if it originally comes from FreeBSD.

Among other utilities, it installs the "bsdtar" program, which you can use in your scripts like this:

  bsdtar --create --verbose --format=pax --file="${DIRECTORY}".pax "${DIRECTORY}" || exit
And for extraction:

  bsdtar --extract --preserve-permissions --verbose --file="${DIRECTORY}".pax
The bsdtar program has options for compressing and/or encrypting the archives, for the case when you do not want to use directly other external programs.

"par2create" creates multiple files from the (normally compressed and encrypted) archive file, for storing the added redundancy. I make a directory where I move those files, then I use a second time bsdtar (obviously without any compression or encryption) to aggregate those files in a single archive with redundancy.

The libarchive package can also be taken directly from:

https://github.com/libarchive/libarchive

"libarchive" handles correctly all kinds of file metadata, e.g. extended file attributes and high-resolution file timestamps, which not all archiving utilities do. Many Linux utilities, with the default command-line options or when they have not been compiled from their source with adequate compilation options, which happens in some Linux distributions, may silently lose some of the file metadata, when copying, moving or archiving.

Re: Unpowered SSDs slowly lose data

#312

Post by adrian_b »

xhkkffbf wrote: post #262
mrspuratic wrote: post #228

Earlier quoted context omitted.

CD storage has an interesting take, the available sector size varies by use, i.e. audio or MPEG1 video (VideoCD) at 2352 data octets per sector (with two media level ECCs), actual data at 2048 octets per sector where the extra EDC/ECC can be exposed by reading "raw". I learned this the hard way with VideoPack's malformed VCD images, I wrote a tool to post-process the images to recreate the correct EDC/ECC per sector.…

Octets? Don't you mean "bytes"? Or is that word problematic now?

Octets is the term used in most international standards instead of the American "byte".

"Octet" has the advantage that it is not ambiguous. In old computer documentation, from the fifties to the late sixties, a "byte" could have meant any size between 6 bits and 16 bits, the same like "word", which could have meant anything between 8 bits and 64 bits, including values like 12 bits, 18 bits, 36 bits, 60 bits, or even 43 bits.

Traditionally, computer memory is divided in pages, which are divided in lines, which are divided in words, which are divided in bytes. However the sizes of any of those "units" has varied in very wide ranges in the early computers.

IBM System/360 has chosen the 8-bit byte, and the dominance of IBM has then forced this now ubiquitous meaning of "byte", but there were many computers before System/360 and many coexisting for some years with the IBM 360 and later mainframes, where byte meant something else.

Re: Unpowered SSDs slowly lose data

#313

Post by paranoidrobot »

For years I've been collecting a slowly growing pile of old drives from old devices as I had replaced them.

I finally decided to do something about them and got myself a USB to IDE/SATA adapter and spent a week or so duplicating them so I could check if there was anything worthwhile to keep, before destroyig and sending them to an e-waste recycler.

Some of those drives had been sitting around since early 2000.

All except two of the mechanical drives just fired up and worked fine. One won't power on at all. One powers up, but reading from some sectors just results in failures.

But the SSDs are another story. All blank/empty for any that were more than a few years old. Even though I know some were pulled from working machines.

Re: Unpowered SSDs slowly lose data

#314

Post by lazide »

mort96 wrote: post #272
lazide wrote: post #255

Earlier quoted context omitted.

Yes, and? HDD controllers dying and head crashes are a thing too. At least in the ‘bricked’ case it’s a trivial RMA - corrupt blocks tend to be a harder fight. And since ‘bricked’ is such a trivial RMA, manufacturers have more of an incentive to fix it or go broke, or avoid it in the first place. This is why backups are important now; and always have been.

We're not talking about the SSD controller dying. The SSD controller in the hypothetical situation that's being described is working as intended.

Not as far as I can tell, where intended is ‘as any user would reasonably expect’. Bricking the drive (can’t even read) because of too many errors is not what most users would ever want.

Some would (enterprise maybe), but even then they’d want deterministic data deletes too, which doesn’t sound like are happening.

Re: Unpowered SSDs slowly lose data

#315

Post by compsciphd »

adrian_b wrote: post #311
casenmgreen wrote: post #293

Earlier quoted context omitted.

Thank you for this. I had no knowledge of pax, or that par was an open standard, and I care about what they help with. Going to switch over to using both in my backups.

For handling pax archives, I recommend the "libarchive" package, which is available in many Linux distributions, even if it originally comes from FreeBSD. Among other utilities, it installs the "bsdtar" program, which you can use in your scripts like this: bsdtar --create --verbose --format=pax --file="${DIRECTORY}".pax "${DIRECTORY}" || exit And for extraction: bsdtar --extract --preserve-permissions --verbose --fil…

there's no reason that you have to create multiple files for par2 if you are storing the recovery data with the protected data. It only was split into files of varying size due to its source in protecting usenet posted binaries to allow users to not have to download the entire recovery data when they only needed a portion.

Re: Unpowered SSDs slowly lose data

#317

Post by bzzzt »

immibis wrote: post #298
cogman10 wrote: post #289

Earlier quoted context omitted.

I really wish this responsibility was something hoisted up into the FS and not a responsibility of the drive itself. It's ridiculous (IMO) that SSD firmware is doing so much transparent work just to keep the illusion that the drive is actually spinning metal with similar sector write performance.

Linux supports raw flash, called an MTD device (memory technology device). It's often used in embedded systems. And it has MTD-native filesystems such as ubifs. But it's only really used in embedded systems because... PC SSDs don't expose that kind of interface. (Nor would you necessarily want them to. A faulty driver would quietly brick your hardware in a matter of minutes to hours)

A buggy firmware will brick an SSD and block every option for recovering at least part of the data.

Re: Unpowered SSDs slowly lose data

#318

Post by bzzzt »

cogman10 wrote: post #289
jeremyvisser wrote: post #218

Earlier quoted context omitted.

Here’s the thing. That SSD controller is the interface between you and those blocks. If it decides, by some arbitrary measurement, as defined by some logic within its black box firmware, that it should stop returning all blocks, then it will do so, and you have almost no recourse. This is a very common failure mode of SSDs. As a consequence of some failed blocks (likely exceeding a number of failed blocks, or perhaps…

I really wish this responsibility was something hoisted up into the FS and not a responsibility of the drive itself. It's ridiculous (IMO) that SSD firmware is doing so much transparent work just to keep the illusion that the drive is actually spinning metal with similar sector write performance.

Seems like the approach Apple is taking by soldering storage directly on the mainboard or using proprietary modules like in the Mac mini.

Re: Unpowered SSDs slowly lose data

#319

Post by mrspuratic »

xhkkffbf wrote: post #262
mrspuratic wrote: post #228

Earlier quoted context omitted.

CD storage has an interesting take, the available sector size varies by use, i.e. audio or MPEG1 video (VideoCD) at 2352 data octets per sector (with two media level ECCs), actual data at 2048 octets per sector where the extra EDC/ECC can be exposed by reading "raw". I learned this the hard way with VideoPack's malformed VCD images, I wrote a tool to post-process the images to recreate the correct EDC/ECC per sector.…

Octets? Don't you mean "bytes"? Or is that word problematic now?

Not problematic, minor pedantry. With much time spent reading (and occasionally writing) technical documentation it's octets, binary prefixes, and other wanton pedantry where likely to be understood/appreciated or precision is required.

FTR, ECMA-130 (the CD "yellow book" equivalent standard) is littered with the term "8-bit bytes", so it was certainly a thing then. Precision when simultaneously discussing eight-to-fourteen modulation, and the 17 encoding "bits" that hit the media for each octet as noted in a sibling comment.

Now, woktets on the other hand...

Re: Unpowered SSDs slowly lose data

#320

Post by mort96 »

lazide wrote: post #314
mort96 wrote: post #272

Earlier quoted context omitted.

We're not talking about the SSD controller dying. The SSD controller in the hypothetical situation that's being described is working as intended.

Not as far as I can tell, where intended is ‘as any user would reasonably expect’. Bricking the drive (can’t even read) because of too many errors is not what most users would ever want. Some would (enterprise maybe), but even then they’d want deterministic data deletes too, which doesn’t sound like are happening.

You can argue that controllers shouldn't behave that way. But they do, it's not a bug, and it's not a dead controller. It's a perfectly functional controller's response to dead blocks.
Post reply on HN