Xz format inadequate for long-term archiving (2016) • Hacker News Board

Re: Xz format inadequate for long-term archiving (2016)

#21

Post by esaym » Wed, Jun 05, 2019, 5:01 PM UTC

gmueckl wrote: post #20

esaym wrote: post #9

Interestingly, since "recovery" is mentioned several times, I decided to test myself. I took a copy of a jpeg image, compressed it different times with either gzip or bzip2, then with a hexeditor modified one byte. The recovery instructions for gzip is to simply do "zcat corrupt_file.gz > corrupt_file". While for bzip2 is to use the bzip2recover command which just dumps the blocks out individually (corrupt ones and a…

Your method is clearly flawed. Altering a single byte once is insufficient as a test unless you analyzed the structure of the compressed file first to see where the really important information is stored. It may well be that you just modified a verbatim string from the source data in the gzip case, but corrupted a bit of metadata about how the compressed data is structured in the bzip2 case. If you tried a different…

I thought about that as well. I tried it three different times all with the same results.

Re: Xz format inadequate for long-term archiving (2016)

#22

Post by AdmiralAsshat » Wed, Jun 05, 2019, 5:01 PM UTC

jordigh wrote: post #8

This guy used to go around GNU mailing lists (and others) trying to get us to use lzip. https://gcc.gnu.org/ml/gcc/2017-06/msg00044.html https://lists.debian.org/debian-devel/2017/06/msg00433.html It was a bit bizarre when he hit the Octave mailing list. Eventually, people just wanted xz back: http://octave.1599824.n4.nabble.com/opinion-bring-back-Octav...

Everyone's gotta have their white whale, I guess.

Re: Xz format inadequate for long-term archiving (2016)

#23

Post by matt-attack » Wed, Jun 05, 2019, 5:09 PM UTC

xoa wrote: post #13

Not that many of the complaints aren't reasonable, but I thought that in general compression/format was orthogonal to parity, which is what I assume is actually wanted for long-term archiving? I always figured that the goal should normally to be able to get back out a bit-perfect copy of whatever went in, using something like Parchive at the file level or ZFS for online storage at the fs level. I guess on the princip…

I think with archiving it’s more than that. Sure you can guarantee that the actual tool you just compressed with can restore the original perfectly. But with long term digital archiving I think you need the assurance that the “spec” called “xz” could be perfectly reimplemented by an expert in the future. Based solely on documentation. And on a platform that doesn’t exist today. That is, you must assume the original executable is either not available or not able to be executed.

Re: Xz format inadequate for long-term archiving (2016)

#24

Post by mixmastamyk » Wed, Jun 05, 2019, 5:12 PM UTC

asveikau wrote: post #16

StavrosK wrote: post #12

Earlier quoted context omitted.

Why are you assuming they aren't assuming redundant storage? Redundant storage isn't a cure-all, there's still a chance two blocks on two disks will fail in the exact same spot.

Seems easier to increase the amount of disks and address it at a low layer than to re-engineer all layers, all file formats, for corruption.

If there are safer free formats, why not use them? It's not like all data everywhere is always going to be stored on zfs.

Re: Xz format inadequate for long-term archiving (2016)

#25

Post by gwern » Wed, Jun 05, 2019, 5:19 PM UTC

mixmastamyk wrote: post #24

asveikau wrote: post #16

Earlier quoted context omitted.

Seems easier to increase the amount of disks and address it at a low layer than to re-engineer all layers, all file formats, for corruption.

If there are safer free formats, why not use them? It's not like all data everywhere is always going to be stored on zfs.

By the 'end to end principle', redundancy should probably be concentrated somewhere in the stack, and the rest of the stack should be concerned merely with validating integrity. It's unlikely that the optimum balance of resources and loss probability will entail redundancy at every level of the stack, from raw HDD bytes up to the global system level.

Re: Xz format inadequate for long-term archiving (2016)

#26

Post by zmix » Wed, Jun 05, 2019, 5:27 PM UTC

nkoren wrote: post #11

Thoreandan wrote: post #5

Earlier quoted context omitted.

and "from the author of lzip, a competing lzma library that never went viral". Welcome to the Better Technology that Shoulda Made It bench. Your seat's over there next to OS/2, BeOS, and OpenGenera.

Amiga forever!!!!!!

Ha! I just wanted to add this. But you did it first! :-)

Re: Xz format inadequate for long-term archiving (2016)

#27

Post by h1d » Wed, Jun 05, 2019, 5:31 PM UTC

microcolonel wrote: post #6

> "3 Then, why some free software projects use xz?" Because the files are usually smaller than gzip, with faster decompression than bzip2, and the library is available on most systems.

Archiving for distribution and backups are very different things. You don't care if some app distribution compressed file gets corrupted, you just compress again but your compressed backup files usually don't have much source of reference.

I wouldn't use any unreliable format for backups. I picked bzip2 for stability and compression rate.

Re: Xz format inadequate for long-term archiving (2016)

#28

Post by microcolonel » Wed, Jun 05, 2019, 5:35 PM UTC

h1d wrote: post #27

microcolonel wrote: post #6

> "3 Then, why some free software projects use xz?" Because the files are usually smaller than gzip, with faster decompression than bzip2, and the library is available on most systems.

Archiving for distribution and backups are very different things. You don't care if some app distribution compressed file gets corrupted, you just compress again but your compressed backup files usually don't have much source of reference. I wouldn't use any unreliable format for backups. I picked bzip2 for stability and compression rate.

In my opinion, the compressor is not the right place to add data integrity mechanisms, especially since data integrity mechanisms only really apply to particular media. Data on hard drives don't get corrupted in the same way as data on TLC SSDs, and generally on the latter you're better off with redundancy and diversification, than with inline error correcting codes.

Honestly, I don't see why xz should have any of its own data integrity mechanisms whatsoever, except maybe a whole-archive CRC32 or similar.

Re: Xz format inadequate for long-term archiving (2016)

#29

Post by xoa » Wed, Jun 05, 2019, 5:37 PM UTC

matt-attack wrote: post #23

xoa wrote: post #13

Not that many of the complaints aren't reasonable, but I thought that in general compression/format was orthogonal to parity, which is what I assume is actually wanted for long-term archiving? I always figured that the goal should normally to be able to get back out a bit-perfect copy of whatever went in, using something like Parchive at the file level or ZFS for online storage at the fs level. I guess on the princip…

I think with archiving it’s more than that. Sure you can guarantee that the actual tool you just compressed with can restore the original perfectly. But with long term digital archiving I think you need the assurance that the “spec” called “xz” could be perfectly reimplemented by an expert in the future. Based solely on documentation. And on a platform that doesn’t exist today. That is, you must assume the original e…

I don't disagree with that at all! I did try to say (maybe unclearly) that having simpler more foolproof and failsafe layers at every level seems absolutely worth pursuing anyway, where possible. But I also wonder whether some of the common wisdom is from an age that is obsolete for many scenarios? Ie., in the 80s and 90s and even early 00s there was a lot more churn, practices were less standardized, computing time was more expensive, storage capacity was far more expensive, the ratio of software to data size was higher, etc. The latter seems to tie into "must assume the original executable is either not available or not able to be executed." For serious archiving, does it no make sense to just bundle in not merely the executable, but in some instances an entire environment? In a "fall of civilization" type scenario that may not be helpful vs a clear simple spec and ease of bootstrapping, but for situations where technological continuity is a limiting factor for some reason anyway is it safe to simply assume basics like at this point x86 will never go away as something that is at least virtualized? In my own experience there is a pretty clear cutoff date after which I can continue to run the entire environment in a VM.

Again this is shooting the breeze a bit, article is discussing a case where there should be the freedom to choose better formats. But for a lot of important archive material, including software itself, are we getting to the point where many long term archives should simply including everything necessary to deal with them in the present day as a container or VM image, which is then stored with a solid amount of parity and replication?

Re: Xz format inadequate for long-term archiving (2016)

#30

Post by LinuxBender » Wed, Jun 05, 2019, 5:37 PM UTC

zamalek wrote: post #18

LinuxBender wrote: post #3

If you first use tar to preserve xattrs/etc.. then you can use anything to compress. xz, bz2, 7z, even arj if you are feeling nostalgic. tar cvfJ ./files.tar.xz /some/dir

You've missed the point of the article entirely. A single bit-flip (which is almost guaranteed over long-term ) can easily render the entire xz file corrupt. This has nothing to do with xattrs/etc.

Yes, I am totally on auto-pilot today. I'm used to a different article that gets re-posted often about xz and my browser blocks non-https sites so I assumed it was that other article.

That said, I use xz in automation that compresses files on one end and decompresses on the other. I've not had any file corruption thus far. checksums always match. Hopefully the author has submitted bug reports and ways to reproduce.