Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Xz format inadequate for long-term archiving (2017)

nongnu.org

Post reply on HN
Print view View original

101–110 of 138 posts

Re: Xz format inadequate for long-term archiving (2017)

#101

Post by viraptor »

rikkus wrote: post #62
jlgaddis wrote: post #45

Earlier quoted context omitted.

> So you've archived two or more copies of each file You haven't? It took me just one minor "data loss incident" ~20 years ago to very quickly convince me to become a lifetime member of the "backup all the things to a few different locations" club. > That means you're use at least twice as much space (and if you're keeping the original as well, more than twice). "Storage is cheap."

Storage is cheap indeed, though it takes some effort to make it cheap. 99% of the digital data I'm keeping for the long term is family photos and videos. All my photos go to Dropbox (easy copy-from-device and access anywhere) and are then backed up to multiple locations by CrashPlan. It'll be a while yet, but in the next few years I'll be hitting the 1TB Dropbox limit. I'm hoping that Dropbox make a >1TB 'consumer' p…

> Storage might be cheap, but keeping it cheap is tricky.

If it's really for pure backup, not continuous sync, Glacier is $4 per TB.

Re: Xz format inadequate for long-term archiving (2017)

#102

Post by pas »

blattimwind wrote: post #59
pas wrote: post #55

Earlier quoted context omitted.

For small scale, use Dropbox or Google Drive, or whatever, because for small scale the most important part of backup is actually reliably having it done. If you rely on manual process, you're doomed. :) For large scale in house things: Ceph regurarly does scrubbing of the data. (Compares checksums.) and DreamHost has DreamObjects. Thanks for mentioning borg/restic, I have never heard of them. (rsnapshot [rsync] works…

I think rsync/rsnapshot aren't really appropiate for backups: (1) They need full support for all FS oddities (xattrs, rforks, acls etc.) wherever you move the data (2) They don't checksum the data at all. The newer tools don't have either problem that much: For (1) they pack/unpack these in their own format which doesn't need anything special, so if you move your data twice in a circle you won't lose any (but their s…

Hm, rsync does MD5 checking automatically. Which doesn't do much against bitrot [0], but it should help with the full circle thing. (And maybe it'll be SHA256+ in newer versions? Though there's not even a ticket in their bugzilla about this. And maybe MD5 is truly enough against random in-transit corruption.)

Yeah, crypto is something that doesn't play well with dedupe, especially if you don't trust the target backup server.

Uh, BREACH was a beast (he-he). I'm a bit still uneasy after thinking about how long these bugs were lurking in OpenSSL. Thankfully the splendid work of Intel engineers quickly diverted the nexus of our bad feels away from such high level matters :|

[0] That's something that the btrfs/ZFS/Ceph should/could fix. (And btrfs supports incremental mode for send+receive.)

Re: Xz format inadequate for long-term archiving (2017)

#103

Post by imtringued »

viraptor wrote: post #101
rikkus wrote: post #62

Earlier quoted context omitted.

Storage is cheap indeed, though it takes some effort to make it cheap. 99% of the digital data I'm keeping for the long term is family photos and videos. All my photos go to Dropbox (easy copy-from-device and access anywhere) and are then backed up to multiple locations by CrashPlan. It'll be a while yet, but in the next few years I'll be hitting the 1TB Dropbox limit. I'm hoping that Dropbox make a >1TB 'consumer' p…

> Storage might be cheap, but keeping it cheap is tricky. If it's really for pure backup, not continuous sync, Glacier is $4 per TB.

Glacier is good for dumping data into it but it's absolutely terrible for getting your data out and for full retrievals it's also very expensive. Don't rely on it for anything other than emergency backups of your backups.

Re: Xz format inadequate for long-term archiving (2017)

#104

Post by Symbiote »

pmoriarty wrote: post #69
jlgaddis wrote: post #68

Earlier quoted context omitted.

> Are you really buying an extra 2 TB's of storage to archive 1 TB of data ... $ sudo zpool get size zdata NAME PROPERTY VALUE SOURCE zdata size 21.8T - Yep. It's fine that you "feel secure" with your current backup regimen -- and I certainly hope you never lose any important data. After losing data once, though, I promised myself I'd do my best to make sure that it never happened again. The "primary copy" of all my…

Good for you. If you can afford it, go for it. But I'd still use something like par2 over each of your backups.

I use PAR2, even with multiple copies at different sites, because I look at my photos so rarely that I wouldn't notice a master file had become corrupt before it had mirrored to the other places and the original versions expired (1 year).

5% parity archives is an easy sell, on top of 200% for off site copies.

Re: Xz format inadequate for long-term archiving (2017)

#105

Post by Symbiote »

dv_dt wrote: post #20
snuxoll wrote: post #8

Earlier quoted context omitted.

For long-term archival I think relying on your compression software to protect data integrity is a fool's errand, protecting against bit-rot should be a function of your storage layer as long as you have control over it (in contrast to say, Usenet, where multiple providers have copies of data and you can't trust them to not lose part of it - hence the inclusion of .par files for everything under alt.binaries).

I keep seeing recommendations for par/par2 but it seems like as software, the project isn't actively maintained? As an aside, that makes me think of dead languages and the use of latin for scientific names because it isn't changing anymore... but do you want that out of archival formats and software?

It's probable that PAR2 is essentially feature complete, so no maintenance is really needed.

The program does pretty much the same thing as it did a decade ago.

Re: Xz format inadequate for long-term archiving (2017)

#106

Post by yorwba »

jwilliams wrote: post #76
londons_explore wrote: post #52

Earlier quoted context omitted.

If you get compression ratios that good, you should consider if your application might be doing something stupid like storing the same data thousands of times inside it's data file. If you store enough of the same type of data, invest in redesigning the application. There's a reason we all use jpegs over zipped bitmaps...

> There's a reason we all use jpegs over zipped bitmaps... It's because it's an appropriate compression - just like xz can be? Not sure what you're actually suggesting here.

The suggestion is to design an application-specific format that avoids storing redundant data in the first place. When that's an option at all it gives you higher compression than any general-purpose compression algorithm can achieve.

Re: Xz format inadequate for long-term archiving (2017)

#107

Post by white-flame »

viraptor wrote: post #101
rikkus wrote: post #62

Earlier quoted context omitted.

Storage is cheap indeed, though it takes some effort to make it cheap. 99% of the digital data I'm keeping for the long term is family photos and videos. All my photos go to Dropbox (easy copy-from-device and access anywhere) and are then backed up to multiple locations by CrashPlan. It'll be a while yet, but in the next few years I'll be hitting the 1TB Dropbox limit. I'm hoping that Dropbox make a >1TB 'consumer' p…

> Storage might be cheap, but keeping it cheap is tricky. If it's really for pure backup, not continuous sync, Glacier is $4 per TB.

That's $4 per TB-month. Meaning you're effectively paying more than the cost of a 1TB hard drive replaced every year, for every TB you're storing. Plus fees to get your data back out. An 8TB drive, replaced every year, is half the cost per TB, with no additional access cost.

Depending on how price conscious you are, I agree with the GP's "keeping it cheap is tricky". And with things like backup, even if you do it yourself, the time spent maintaining it should be negligible: Occasionally kick off a format shift or failed drive replacement, have scripts running everything else.

Re: Xz format inadequate for long-term archiving (2017)

#108

Post by arghwhat »

moltensyntax wrote: post #84

This article again? In my opinion, this article is biased. The subtext here is that the author is claiming that his "lzip" format is superior. But xz was not chosen "blindly" as the article claims. To me, most of the claims are arguable. To say 3 levels of headers is "unsafe complexity"... I don't agree. Indirection is fundamental to design. To say padding is "useless"... I don't understand why padding and byte-align…

> To say 3 levels of headers is "unsafe complexity"... I don't agree. Indirection is fundamental to design.

3 individual headers for one file format is unnecessary complexity.

> To say padding is "useless"

Padding in general is not useless, but padding in a compression format is very counterproductive.

> And it's pure FUD to say "Xz implementations may choose what subset of the format they support. They may even choose to not support integrity checking at all. Safe interoperability among xz implementations is not guaranteed". You could say this about any software - "oh no, someone might make a bad implementation!" Format fragmentation is essentially a social problem more than a technical problem.

This isn't about "someone making a bad implementation!", it's about crucial features being optional. That is, completely compliant implementations may or may not be able to decompress a given XZ archive, and may or may not be able to validate the archive.

XZ may not have been chosen blindly, but it certainly does not seem like a sensible format. There is no benefit to this complexity. We do not need or benefit from a format that is flexible, as we can just swap format and tool if we want to swap algorithms, like we have done so many times before (a proper compression format is just a tiny algorithm-specific header + trailing checksum, so it is not worth generalizing away).

Any and all benefits of XZ lie in LZMA2. We could have lzip2 and avoid all of these problems.

(I have no opinion as to whether LZIP should supersede GZIP/BZIP2, but XZ certainly seems like a poor choice.)

Re: Xz format inadequate for long-term archiving (2017)

#109

Post by comex »

Last time this came up on HN, I did some research, and discovered that lzip was quite non-robust in the face of data corruption: a single bit flip in the right place in an lzip archive could cause the decompressor to silently truncate the decompressed data, without reporting an error. Not only that, this vulnerability was a direct consequence of one of the features used to claim superiority to XZ: namely, the ability to append arbitrary “trailing data” to an lzip archive without invalidating it.

Like some other compressed formats, an lzip file is just a series of compressed blocks concatenated together, each block starting with a magic number and containing a certain amount of compressed data. There’s no overall file header, nor any marker that a particular block is the last one. This structure has the advantage that you can simply concatenate two lzip files, and the result is a valid lzip file that decompresses to the concatenation of what the inputs decompress to.

Thus, when the decompressor has finished reading a block and sees there’s more input data left in the file, there are two possibilities for what that data could contain. It could be another lzip block corresponding to additional compressed data. Or it could be any other random binary data, if the user is taking advantage of the “trailing data” feature, in which case the rest of the file should be silently ignored.

How do you tell the difference? Simply enough, by checking if the data starts with the 4-byte lzip magic number. If the magic number itself is corrupted in any way? Then the entire rest of the file is treated as “trailing data” and ignored. I hope the user notices their data is missing before they delete the compressed original…

It might be possible to identify an lzip block that has its magic number corrupted, e.g. by checking whether the trailing CRC is valid. However, at least at the time I discovered this, lzip’s decompressor made no attempt to do so. It’s possible the behavior has improved in later releases; I haven’t checked.

But at least at the time this article was written: pot, meet kettle.

Re: Xz format inadequate for long-term archiving (2017)

#110

Post by viraptor »

white-flame wrote: post #107
viraptor wrote: post #101

Earlier quoted context omitted.

> Storage might be cheap, but keeping it cheap is tricky. If it's really for pure backup, not continuous sync, Glacier is $4 per TB.

That's $4 per TB- month . Meaning you're effectively paying more than the cost of a 1TB hard drive replaced every year, for every TB you're storing. Plus fees to get your data back out. An 8TB drive, replaced every year, is half the cost per TB, with no additional access cost. Depending on how price conscious you are, I agree with the GP's "keeping it cheap is tricky". And with things like backup, even if you do it y…

> Meaning you're effectively paying more than the cost of a 1TB hard drive replaced every year, for every TB you're storing.

Yes. But what you get in return is not having that data at home. It doesn't matter how many copies you have locally if your home gets robbed, flooded, or burns down.

Post reply on HN