Use par2 to generate FEC for your archives and move on with your life.
Xz format inadequate for long-term archiving (2017)
121–130 of 138 posts
Re: Xz format inadequate for long-term archiving (2017)
#122This article again? In my opinion, this article is biased. The subtext here is that the author is claiming that his "lzip" format is superior. But xz was not chosen "blindly" as the article claims. To me, most of the claims are arguable. To say 3 levels of headers is "unsafe complexity"... I don't agree. Indirection is fundamental to design. To say padding is "useless"... I don't understand why padding and byte-align…
> To say 3 levels of headers is "unsafe complexity"... I don't agree. Indirection is fundamental to design. 3 individual headers for one file format is unnecessary complexity. > To say padding is "useless" Padding in general is not useless, but padding in a compression format is very counterproductive. > And it's pure FUD to say "Xz implementations may choose what subset of the format they support. They may even choo…
> 3 individual headers for one file format is unnecessary complexity.
So all these file formats are unnecessarily complex?
- all OpenDocument formats
- all MS office formats
- all multimedia container formats
- deb/rpm packages
etc?
Re: Xz format inadequate for long-term archiving (2017)
#123Earlier quoted context omitted.
> To say 3 levels of headers is "unsafe complexity"... I don't agree. Indirection is fundamental to design. 3 individual headers for one file format is unnecessary complexity. > To say padding is "useless" Padding in general is not useless, but padding in a compression format is very counterproductive. > And it's pure FUD to say "Xz implementations may choose what subset of the format they support. They may even choo…
> 3 individual headers for one file format is unnecessary complexity. So all these file formats are unnecessarily complex? - all OpenDocument formats - all MS office formats - all multimedia container formats - deb/rpm packages etc?
> rpm packages
I recall an article posted here detailing how incredibly bloated and crufty the RPM format was.
Re: Xz format inadequate for long-term archiving (2017)
#124This article again? In my opinion, this article is biased. The subtext here is that the author is claiming that his "lzip" format is superior. But xz was not chosen "blindly" as the article claims. To me, most of the claims are arguable. To say 3 levels of headers is "unsafe complexity"... I don't agree. Indirection is fundamental to design. To say padding is "useless"... I don't understand why padding and byte-align…
"Look at how much padding the tar format has. And tar is a good example of how "useless padding" was used to extend the format to support larger files. So this supposed "flaw" has been in tar for dozens of years, with no disastrous effects at all." Just because it's in tar doesn't mean that the design is flawless. tar was created a long time ago, when a lot of things we are concerned with now weren't even thought of.…
Thanks for such an amazing rabbit-hole of a link.
Re: Xz format inadequate for long-term archiving (2017)
#125Earlier quoted context omitted.
> To say 3 levels of headers is "unsafe complexity"... I don't agree. Indirection is fundamental to design. 3 individual headers for one file format is unnecessary complexity. > To say padding is "useless" Padding in general is not useless, but padding in a compression format is very counterproductive. > And it's pure FUD to say "Xz implementations may choose what subset of the format they support. They may even choo…
> 3 individual headers for one file format is unnecessary complexity. So all these file formats are unnecessarily complex? - all OpenDocument formats - all MS office formats - all multimedia container formats - deb/rpm packages etc?
Aren't MS office formats the poster child for overly complex file formats?
Re: Xz format inadequate for long-term archiving (2017)
#126Earlier quoted context omitted.
Storage is cheap indeed, though it takes some effort to make it cheap. 99% of the digital data I'm keeping for the long term is family photos and videos. All my photos go to Dropbox (easy copy-from-device and access anywhere) and are then backed up to multiple locations by CrashPlan. It'll be a while yet, but in the next few years I'll be hitting the 1TB Dropbox limit. I'm hoping that Dropbox make a >1TB 'consumer' p…
Are you using the small business version of CrashPlan? I was using CP too until they discontinued their B2C.
Same here - I'm currently still migrating from CrashPlan B2C to using Arq backing up to Backblaze B2. (Being able to access B2 from Panic's Transmit Mac app made B2 really attractive to me as well, and it looks like I'll save a lot of money compared with CrashPlan.)
Re: Xz format inadequate for long-term archiving (2017)
#127I upvoted this because it seems to make some good points and I think the topic is interesting and important, but I can't understand why the "Then, why some free software projects use xz?" section does not mention xz's main selling point of being better than other commonly used alternatives at compressing things to smaller sizes. https://www.rootusers.com/gzip-vs-bzip2-vs-xz-performance-co...
> compressing things to smaller sizes. ...relative to ... ? Is it better than lzip? lzip sounds like it would also use LZMA-based compression, right? This [1] sounds like an interesting and more detailed/up-to-date comparison. Also by the same author BTW. [1] https://www.nongnu.org/lzip/lzip_benchmark.html#xz
I did some tests of my own and xz turned out marginally better than lzip in most of them.
665472 freebsd-11.0-release-amd64-disc1.iso
401728 freebsd-11.0-release-amd64-disc1.iso.xz 5m0.606s
406440 freebsd-11.0-release-amd64-disc1.iso.lz 5m43.375s
430872 freebsd-11.0-release-amd64-disc1.iso.bz2 1m38.654s
440400 freebsd-11.0-release-amd64-disc1.iso.gz 0m27.073s
431740 freebsd-11.0-release-amd64-disc1.iso.zst 0m3.424s
Re: Xz format inadequate for long-term archiving (2017)
#128Last time this came up on HN, I did some research, and discovered that lzip was quite non-robust in the face of data corruption: a single bit flip in the right place in an lzip archive could cause the decompressor to silently truncate the decompressed data, without reporting an error. Not only that, this vulnerability was a direct consequence of one of the features used to claim superiority to XZ: namely, the ability…
The maintainer's response when I reported this bug was 'Just use "lzip -vvvv" to see the warning':
https://lists.debian.org/55C0FE82.7050700@gnu.org
Their advocacy in this thread was so good that I removed lzip from my system.
Re: Xz format inadequate for long-term archiving (2017)
#129So long as xz(1) gets insane amounts of compression and there is no compressor which compresses better, people are going to keep preferring it.
Re: Xz format inadequate for long-term archiving (2017)
#130I upvoted this because it seems to make some good points and I think the topic is interesting and important, but I can't understand why the "Then, why some free software projects use xz?" section does not mention xz's main selling point of being better than other commonly used alternatives at compressing things to smaller sizes. https://www.rootusers.com/gzip-vs-bzip2-vs-xz-performance-co...
You can see discussion of this point, from the last time that this article was on Hacker News, at https://news.ycombinator.com/item?id=12769277 .