Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Xz format inadequate for long-term archiving (2017)

nongnu.org

Post reply on HN
Print view View original

41–50 of 138 posts

Re: Xz format inadequate for long-term archiving (2017)

#41

Post by kazinator »

zzzcpan wrote: post #30
kazinator wrote: post #22

> The xz format lacks a version number field. The only reliable way of knowing if a given version of a xz decompressor can decompress a given file is by trial and error. Wow ... that is inexcusably idiotic. Whoever designed that shouldn't be programming. Out of professional disdain, I pledge never to use this garbage.

Welcome to the world of software I guess. Non idiotic things are rare here.

Not w.r.t. that level of idiotic; and in FOSS, at least, we should be able to eject the idiotic. Thanks in part to articles like this, we can.

Re: Xz format inadequate for long-term archiving (2017)

#42

Post by pmoriarty »

tedunangst wrote: post #29

Are these concerns, about error recovery, outdated? If I want to recover a corrupted file, I find another copy. I don't fiddle with the internal length field to fix framing issues. Certainly, if I want to detect corruption, I use a sha256 of the entire file. If that fails, I don't waste time trying to find the bad bit. To add to that, if you need parity to recover from errors, you need to calculate how much based on…

"If I want to recover a corrupted file, I find another copy."

So you've archived two or more copies of each file? That means you're use at least twice as much space (and if you're keeping the original as well, more than twice).

For the likely corruption of the occasional single bit flip here and there, you could do a lot better by using something like par2 and/or dvdisaster (depending on what media you're archiving to).

Re: Xz format inadequate for long-term archiving (2017)

#43

Post by blattimwind »

pas wrote: post #40
cpburns2009 wrote: post #3

It may not be a good choice for long-term data storage, but I disagree that it should not be used for data sharing or software distribution. Different use cases have different needs. If you need long-term storage, it's better to avoid lossless compression that can break after minor corruption. You should also be storing parity/ECC data (I don't recall the subtle difference). If you only need short to moderate term st…

> parity/ECC Parity is ECC (which is usually Reed-Solomon, which is just a fancy name for a big set of more equations than data chunk you have, so that's how it adds in redundancy) with 1 bit. Usually you should aim for +20-40% redundancy. Ceph, HDFS and other distributed storage systems implement erasure coding (which is subtly different from error correction coding), which I would recommend for handling backups.

The interesting thing about erasure codes is that you need to checksum your shards independently from the EC itself. If you supply corrupted or wrong shards, you get corrupted data back.

I think for backup (as in small-scale, "fits on one disk") error-correcting codes are not a really good approach, because IME hard disks with one error you notice usually have made many more errors - or will do so shortly. In that case no ECC will help you. If, on the other hand, you're looking at an isolated error, then only very little data is affected (on average).

For example, a bit error in a chunk in a tool like borg/restic will only break that chunk; a piece of a file or perhaps part of a directory listing.

So for these kinds of scenarios "just use multiple backup drives with fully independent backups" is better and simpler.

Re: Xz format inadequate for long-term archiving (2017)

#44

Post by pmoriarty »

davidw wrote: post #28

FWIW, xz is also a memory hog with the default settings. I inherited an embedded system that attempts to compress and send some logs, using xz, and if they're big enough, it blows up because of memory exhaustion.

"xz is also a memory hog with the default settings"

Then why use the default settings?

I tend to use the maximum settings, which are much more of a memory hog, but I have enough memory where that's not an issue.

Just use the settings that are right for you.

Re: Xz format inadequate for long-term archiving (2017)

#45

Post by jlgaddis »

pmoriarty wrote: post #42
tedunangst wrote: post #29

Are these concerns, about error recovery, outdated? If I want to recover a corrupted file, I find another copy. I don't fiddle with the internal length field to fix framing issues. Certainly, if I want to detect corruption, I use a sha256 of the entire file. If that fails, I don't waste time trying to find the bad bit. To add to that, if you need parity to recover from errors, you need to calculate how much based on…

"If I want to recover a corrupted file, I find another copy." So you've archived two or more copies of each file? That means you're use at least twice as much space (and if you're keeping the original as well, more than twice). For the likely corruption of the occasional single bit flip here and there, you could do a lot better by using something like par2 and/or dvdisaster (depending on what media you're archiving t…

> So you've archived two or more copies of each file

You haven't?

It took me just one minor "data loss incident" ~20 years ago to very quickly convince me to become a lifetime member of the "backup all the things to a few different locations" club.

> That means you're use at least twice as much space (and if you're keeping the original as well, more than twice).

"Storage is cheap."

Re: Xz format inadequate for long-term archiving (2017)

#47

Post by jwilliams »

I sent a reasonable amount of data to Cloud Storage. It varies a lot. Usually ~10GB/day, but it can be up to 1TB/day regularly.

xz can be amazing. It can also bite you.

I've had payloads that compress to 0.16 with gzip then compress to 0.016 with xz. Hurray! Then I've had payloads where xz compression is par, or worse. However, with "best or extreme" compression, xz can peg your CPU for much longer. gzip and bzip2 will take minutes and xz -9 is taking hours at 100% CPU.

As annoying as that is, getting an order of magnitude better in many circumstances is hard to give up.

My compromise is "xz -1". It usually delivers pretty good results, in reasonable time, with manageable CPU/Memory usage.

FYI. The datasets are largely text-ish. Usually in 250MB-1GB chunks. So talking JSON data, webpages, and the like.

Re: Xz format inadequate for long-term archiving (2017)

#48

Post by pmoriarty »

When I use xz for archival purposes I always use par2[1] to provide redundancy and recoverability in case of errors.

When I burn data (including xz archives) on to DVD for archival storage, I use dvdisaster[2] for the same purpose.

I've tested both by damaging archives and scratching DVDs, and these tools work great for recovery. The amount of redundancy (with a tradeoff for space) is also tuneable for both.

[1] - https://github.com/Parchive/par2cmdline

[2] - http://dvdisaster.net/

Re: Xz format inadequate for long-term archiving (2017)

#49

Post by niftich »

tzahola wrote: post #39
snuxoll wrote: post #8

Earlier quoted context omitted.

For long-term archival I think relying on your compression software to protect data integrity is a fool's errand, protecting against bit-rot should be a function of your storage layer as long as you have control over it (in contrast to say, Usenet, where multiple providers have copies of data and you can't trust them to not lose part of it - hence the inclusion of .par files for everything under alt.binaries).

Nope. You always need end-to-end parity integrity checking. Your data goes through too many layers before reaching the storage medium. E.g. I once got a substantial amount of my pictures filled with bit errors because of a faulty RAM module in my NAS.

This happened to me, and caused me to rethink my approach file management.

Unfortunately, mainstream tooling is largely fire-and-forget and never includes verification (e.g. copying succeeds even if the written data is getting garbled), so one is forced to use multi-step workflows to get around this. It's pretty discouraging that no strong abstractions exist in this space.

Post reply on HN