Xz format inadequate for long-term archiving (2017) • Hacker News Board

Re: Xz format inadequate for long-term archiving (2017)

#111

Post by kazinator » Fri, Apr 20, 2018, 11:56 PM UTC

comex wrote: post #109

Last time this came up on HN, I did some research, and discovered that lzip was quite non-robust in the face of data corruption: a single bit flip in the right place in an lzip archive could cause the decompressor to silently truncate the decompressed data, without reporting an error. Not only that, this vulnerability was a direct consequence of one of the features used to claim superiority to XZ: namely, the ability…

If the claims in the article are true who cares if the competing thing that the author is working on is also shit (but good to know that too).

Re: Xz format inadequate for long-term archiving (2017)

#112

Post by LeoPanthera » Sat, Apr 21, 2018, 12:38 AM UTC

derefr wrote: post #64

wyldfire wrote: post #26

Earlier quoted context omitted.

> compressing things to smaller sizes. ...relative to ... ? Is it better than lzip? lzip sounds like it would also use LZMA-based compression, right? This [1] sounds like an interesting and more detailed/up-to-date comparison. Also by the same author BTW. [1] https://www.nongnu.org/lzip/lzip_benchmark.html#xz

Relative to the compression formats people were aware of at the time (which didn't include lzip.) People began using xz because mostly because they (e.g. distro maintainers like Debian) had started seeing 7z files floating around, thought they were cool, and so wanted a format that did what 7z did but was an open standard rather than being dictated by some company. xz was that format, so they leapt on it. As it turns…

Can you provide an example of such a .xz file?

Re: Xz format inadequate for long-term archiving (2017)

#113

Post by nebulous1 » Sat, Apr 21, 2018, 12:43 AM UTC

moltensyntax wrote: post #84

This article again? In my opinion, this article is biased. The subtext here is that the author is claiming that his "lzip" format is superior. But xz was not chosen "blindly" as the article claims. To me, most of the claims are arguable. To say 3 levels of headers is "unsafe complexity"... I don't agree. Indirection is fundamental to design. To say padding is "useless"... I don't understand why padding and byte-align…

I feel he has made a case for some inadequacies in Xz. Some of the claims seem exaggerated, such as (2.2) the optional integrity checking, assuming the decompressor at least logs the fact that it couldn't do the integrity checking. Some others are clearly more significant issues, such as (2.5) not checksumming the length fields (2.6) the variable length integers being able to cause framing errors. Others still are petty, such as (2.3) too many possible filters.

While I think he made a case, I somewhat doubt that the other formats are flawless, and the real answer would lie in a more open analysis of all of them.

Re: Xz format inadequate for long-term archiving (2017)

#114

Post by nebulous1 » Sat, Apr 21, 2018, 12:47 AM UTC

tedunangst wrote: post #29

Are these concerns, about error recovery, outdated? If I want to recover a corrupted file, I find another copy. I don't fiddle with the internal length field to fix framing issues. Certainly, if I want to detect corruption, I use a sha256 of the entire file. If that fails, I don't waste time trying to find the bad bit. To add to that, if you need parity to recover from errors, you need to calculate how much based on…

While finding another copy might be a practical solution for most of us, it seems like a wrongheaded way of designing an archiving data format.

Re: Xz format inadequate for long-term archiving (2017)

#115

Post by pmoriarty » Sat, Apr 21, 2018, 2:28 AM UTC

davidw wrote: post #56

pmoriarty wrote: post #44

Earlier quoted context omitted.

"xz is also a memory hog with the default settings" Then why use the default settings? I tend to use the maximum settings, which are much more of a memory hog, but I have enough memory where that's not an issue. Just use the settings that are right for you.

You'd have to ask the guy who wrote the code in the first place. I think he saw "'best' compression" and stopped looking there.

I didn't mean to ask why the defaults are defaults, but rather why anyone would use the defaults rather than settings more appropritate to their use case?

It's not like xz is unable to be lighter on memory, if that's what you want. It's an option setting away.

Re: Xz format inadequate for long-term archiving (2017)

#116

Post by userbinator » Sat, Apr 21, 2018, 2:29 AM UTC

derefr wrote: post #64

wyldfire wrote: post #26

Earlier quoted context omitted.

> compressing things to smaller sizes. ...relative to ... ? Is it better than lzip? lzip sounds like it would also use LZMA-based compression, right? This [1] sounds like an interesting and more detailed/up-to-date comparison. Also by the same author BTW. [1] https://www.nongnu.org/lzip/lzip_benchmark.html#xz

Relative to the compression formats people were aware of at the time (which didn't include lzip.) People began using xz because mostly because they (e.g. distro maintainers like Debian) had started seeing 7z files floating around, thought they were cool, and so wanted a format that did what 7z did but was an open standard rather than being dictated by some company. xz was that format, so they leapt on it. As it turns…

that did what 7z did but was an open standard rather than being dictated by some company.

7z is an open standard, and the SDK is public domain:

https://www.7-zip.org/sdk.html

Re: Xz format inadequate for long-term archiving (2017)

#117

Post by lopmotr » Sat, Apr 21, 2018, 2:42 AM UTC

comex wrote: post #109

Last time this came up on HN, I did some research, and discovered that lzip was quite non-robust in the face of data corruption: a single bit flip in the right place in an lzip archive could cause the decompressor to silently truncate the decompressed data, without reporting an error. Not only that, this vulnerability was a direct consequence of one of the features used to claim superiority to XZ: namely, the ability…

It's that an implementation problem? I would expect a decompressor to warn that there's unidentified trailing data and perhaps dump it out as-is. After all, even if you did put it there on purpose, surely you still want it, not to have it discarded.

Re: Xz format inadequate for long-term archiving (2017)

#118

Post by joveian » Sat, Apr 21, 2018, 3:23 AM UTC

yason wrote: post #98

LinuxBender wrote: post #86

Earlier quoted context omitted.

xz has substantially better compression than gz or bz2, especially if using the flags -9e. You can use all your cores with -T0 or set how many cores to use. I find it to be on par with 7-zip. Perhaps folks are trying to stick with packages that are in their base repo. p7zip is usually outside of the standard base repos.

Substantially is a relative term. There are niche cases but how many people really care, or need to care, about the last bytes that can be compressed? Packing a bunch of files together as .tgz is a quite universal format and compresses most of the redundancy out. It has some pathological cases but those are rare, and for general files it's still in the same ballpark with other compressors. I remember using .tbz2 in t…

I never thought bz2 was worth it over gzip, but xz is much much better in many common cases (particularly text files, but also other things). Source code can often be xz compressed to about half the size as gzip. If you are downloading multiple things at once or a whole operating system or uploading something then even on slower DSL lines it makes a huge difference IMO. I wish more package systems provided deltas.

The only issue I've had with xz is that it doesn't notice if it is not actually compressing the file like other utilities do and then just store the file uncompressed, so if you try to xz a tar file with a bunch of already highly compressed media files then it both takes forever and and you end up with a nontrivially larger file than you started with.

Also, I like that, unlike gzip, xz can sha256 the uncompressed data if you use the -C sha256 option, providing a good integrity check. Yes, I would really like to use a format that doesn't silently decompress incorrect data and I can't understand why the author of this article thinks that is a bad thing. For backups I keep an mtree file inside the tar file with sha512 of each file and then the -C sha256 option to be able to easily test the compressed tar file without needing another file. In some cases I encrypt the txz with the scrypt utility (which stores HMAC-SHA256 of the encrypted data).

Re: Xz format inadequate for long-term archiving (2017)

#119

Post by technion » Sat, Apr 21, 2018, 4:00 AM UTC

pmoriarty wrote: post #42

tedunangst wrote: post #29

Are these concerns, about error recovery, outdated? If I want to recover a corrupted file, I find another copy. I don't fiddle with the internal length field to fix framing issues. Certainly, if I want to detect corruption, I use a sha256 of the entire file. If that fails, I don't waste time trying to find the bad bit. To add to that, if you need parity to recover from errors, you need to calculate how much based on…

"If I want to recover a corrupted file, I find another copy." So you've archived two or more copies of each file? That means you're use at least twice as much space (and if you're keeping the original as well, more than twice). For the likely corruption of the occasional single bit flip here and there, you could do a lot better by using something like par2 and/or dvdisaster (depending on what media you're archiving t…

I would generally suggest you're more likely to corrupt/lose your whole backup than to have one corrupted bitflip not addressed by the filesystem or underlying storage.

Re: Xz format inadequate for long-term archiving (2017)

#120

Post by microcolonel » Sat, Apr 21, 2018, 4:02 AM UTC

Given that there is basically one standard implementation, and virtually nobody has ever had an issue with compatibility with a given file, I don't see how it is "inadequate". Sure, if it's inadequate now, it'll be inadequate if you read it in a decade, but not in any way which would prevent you from reading it.

If your storage fails, maybe you'll have a problem, but you'd have a problem anyway.

Sometimes I feel like genuine technical concerns are buried by the authors being jerks and blowing things way out of proportion. I, for one, tend to lose interest when I hear hyperbolic mudslinging.