Live data from Hacker News

Xz format inadequate for long-term archiving (2017)

nongnu.org

91–100 of 138 posts

Re: Xz format inadequate for long-term archiving (2017)

#91

> The xz format lacks a version number field. The only reliable way of knowing if a given version of a xz decompressor can decompress a given file is by trial and error. Wow ... that is inexcusably idiotic. Whoever designed that shouldn't be programming. Out of professional disdain, I pledge never to use this garbage.

Histrionic reactions don't improve the overall quality of software.

We certainly should have environments where we can tell someone code is shit, it's just silly and counterproductive to then leap to attacks on the abilities on the person behind it.

Re: Xz format inadequate for long-term archiving (2017)

#93

Earlier quoted context omitted.

"You haven't?" No I don't, because it's a waste of space and money. Using par2 and/or dvdisaster I can archive a lot more files on to the same archival media and still get enough redundancy to feel secure. "Storage is cheap." Cheap is relative. Are you really buying an extra 2 TB's of storage to archive 1 TB of data, because that's what you'd need to do to archive 2 copies of each file. That's a huge waste of space a…

If you have any responsibility for data protection I urge you to read literally anything on disaster recovery procedures.

What if he reads his own comment? That would be covered by the admonition to read literally anything.

But my troll aside, I agree. If losing the data would cause you harm or make you sad (losing photos of your kids for example), you definitely need to have multiple backups in multiple locations, ideally controlled by different parties (so one bug on your cloud provider's side doesn't wipe out both of the copies they store for you). I've been burned by this with personal data a few times. The stakes get even higher when you are responsible for someone else's data. If they don't want to pay for the extra storage, make sure they understand the risk involved.

Re: Xz format inadequate for long-term archiving (2017)

#95
This is purely anecdotal and could easily be PEBKAC, but I created a bunch of xz backups years ago and had to access them a couple of years later after a disc died. To my panicked surprise, when trying to unpack them, I was informed that something was wrong (sorry at this point I don't remember what it was). I never did get it working. From that point on I went back to gzip and have not had a problem since. Yes xz packs efficiently, but a tight archive that doesn't inflate is worse than worthless to me.

Re: Xz format inadequate for long-term archiving (2017)

#96
post #94

So what wrong with plain and simple tar c foo | gzip > foo.tar.gz or tar c foo | bzip2 > foo.tar.bz2 Been using these for over 20 years now. Why is is so important to change things especially as this article points out for the worse?!

Better (smaller and/or faster) compression.

Re: Xz format inadequate for long-term archiving (2017)

#97

Are these concerns, about error recovery, outdated? If I want to recover a corrupted file, I find another copy. I don't fiddle with the internal length field to fix framing issues. Certainly, if I want to detect corruption, I use a sha256 of the entire file. If that fails, I don't waste time trying to find the bad bit. To add to that, if you need parity to recover from errors, you need to calculate how much based on…

"If I want to recover a corrupted file, I find another copy." So you've archived two or more copies of each file? That means you're use at least twice as much space (and if you're keeping the original as well, more than twice). For the likely corruption of the occasional single bit flip here and there, you could do a lot better by using something like par2 and/or dvdisaster (depending on what media you're archiving t…

>So you've archived two or more copies of each file?

If youre not keeping more than 1 copy of important files what are you even doing?

Re: Xz format inadequate for long-term archiving (2017)

#98

Why do people use xz anyway? As for me I just use tar.gz when I need to backup a piece of a Linux file system into an universally-compatible archive, zip when I need to send some files to a non-geek and 7z to backup a directory of plain data files for myself. And I dream of the world to just switch to 7z altogether but it is hardly possible as nobody seems interested in adding tar-like unix-specific metadata support…

xz has substantially better compression than gz or bz2, especially if using the flags -9e. You can use all your cores with -T0 or set how many cores to use. I find it to be on par with 7-zip. Perhaps folks are trying to stick with packages that are in their base repo. p7zip is usually outside of the standard base repos.

Substantially is a relative term. There are niche cases but how many people really care, or need to care, about the last bytes that can be compressed?

Packing a bunch of files together as .tgz is a quite universal format and compresses most of the redundancy out. It has some pathological cases but those are rare, and for general files it's still in the same ballpark with other compressors.

I remember using .tbz2 in the turn of the millennium because at the time download/upload times did matter and in some cases it was actually faster to compress with bzip2 and then send over less data.

But DSL broadband pretty much made it not matter any longer: transfers were fast enough that I don't think I've specifically downloaded or specifically created a .tbz2 archive for years. Good old .tgz is more than enough. Files are usually copied in seconds instead of minutes, and really big files still take hours and hours.

None of the compressors really turn a 15-minute download into a 5-minute download consistently. And the download is likely to be fast enough anyway. Disk space is cheap enough that you haven't needed the best compression methods for ages in order to stuff as much data on portable or backup media.

Ditto for p7zip. It has more features and compresses faster and better but for all practical purposes zip is just as good. Eventhough it's slower it won't take more than a breeze to create and transfer, and it unzips virtually everywhere.

Re: Xz format inadequate for long-term archiving (2017)

#99

> The xz format lacks a version number field. The only reliable way of knowing if a given version of a xz decompressor can decompress a given file is by trial and error. Wow ... that is inexcusably idiotic. Whoever designed that shouldn't be programming. Out of professional disdain, I pledge never to use this garbage.

Histrionic reactions don't improve the overall quality of software. We certainly should have environments where we can tell someone code is shit, it's just silly and counterproductive to then leap to attacks on the abilities on the person behind it.

Improving badly designed software that is unnecessary in the first place is foolish; just "rm -rf" and never give it another thought.

Re: Xz format inadequate for long-term archiving (2017)

#100

This article again? In my opinion, this article is biased. The subtext here is that the author is claiming that his "lzip" format is superior. But xz was not chosen "blindly" as the article claims. To me, most of the claims are arguable. To say 3 levels of headers is "unsafe complexity"... I don't agree. Indirection is fundamental to design. To say padding is "useless"... I don't understand why padding and byte-align…

"Look at how much padding the tar format has. And tar is a good example of how "useless padding" was used to extend the format to support larger files. So this supposed "flaw" has been in tar for dozens of years, with no disastrous effects at all." Just because it's in tar doesn't mean that the design is flawless. tar was created a long time ago, when a lot of things we are concerned with now weren't even thought of.…

The name tar comes from Tape ARchive. Lots of padding makes sense when you know that tar was originally used to write files to magnetic tape, which is highly block oriented. The use of tar today as a bundling and distribution format is something of a misapplication, as it lacks features one might want of such a program.
Post reply on HN