Xz format inadequate for long-term archiving (2016) • Hacker News Board

Re: Xz format inadequate for long-term archiving (2016)

#11

Post by nkoren » Wed, Jun 05, 2019, 4:01 PM UTC

Thoreandan wrote: post #5

SEJeff wrote: post #4

This should have (2016) in the title.

and "from the author of lzip, a competing lzma library that never went viral". Welcome to the Better Technology that Shoulda Made It bench. Your seat's over there next to OS/2, BeOS, and OpenGenera.

Amiga forever!!!!!!

Re: Xz format inadequate for long-term archiving (2016)

#12

Post by StavrosK » Wed, Jun 05, 2019, 4:08 PM UTC

asveikau wrote: post #10

I remember seeing this article before. This time the reaction that surges for me is: if you want long-term archiving but don't assume redundant storage, it's not going to go well. Put your long-term archives on ZFS.

Why are you assuming they aren't assuming redundant storage? Redundant storage isn't a cure-all, there's still a chance two blocks on two disks will fail in the exact same spot.

Re: Xz format inadequate for long-term archiving (2016)

#13

Post by xoa » Wed, Jun 05, 2019, 4:08 PM UTC

Not that many of the complaints aren't reasonable, but I thought that in general compression/format was orthogonal to parity, which is what I assume is actually wanted for long-term archiving? I always figured that the goal should normally to be able to get back out a bit-perfect copy of whatever went in, using something like Parchive at the file level or ZFS for online storage at the fs level. I guess on the principle of layers and graceful failure modes it's better if even sub-archives can handle some level of corruption without total failure, and from a long term perspective of implementation independence simpler/better specified is preferable, but that still doesn't seem to substitute for just having enough parity built in to both notice corruption and fully recover from it to fairly extreme levels.

Re: Xz format inadequate for long-term archiving (2016)

#14

Post by kipari » Wed, Jun 05, 2019, 4:17 PM UTC

StavrosK wrote: post #12

asveikau wrote: post #10

I remember seeing this article before. This time the reaction that surges for me is: if you want long-term archiving but don't assume redundant storage, it's not going to go well. Put your long-term archives on ZFS.

Why are you assuming they aren't assuming redundant storage? Redundant storage isn't a cure-all, there's still a chance two blocks on two disks will fail in the exact same spot.

I reckon that the chance of the same two blocks on two different disks failing between ZFS scrubs would be incredibly small.

Re: Xz format inadequate for long-term archiving (2016)

#15

Post by StavrosK » Wed, Jun 05, 2019, 4:21 PM UTC

kipari wrote: post #14

StavrosK wrote: post #12

Earlier quoted context omitted.

Why are you assuming they aren't assuming redundant storage? Redundant storage isn't a cure-all, there's still a chance two blocks on two disks will fail in the exact same spot.

I reckon that the chance of the same two blocks on two different disks failing between ZFS scrubs would be incredibly small.

Yes, over normal timescales. A lot can happen in a thousand years.

Re: Xz format inadequate for long-term archiving (2016)

#16

Post by asveikau » Wed, Jun 05, 2019, 4:29 PM UTC

StavrosK wrote: post #12

asveikau wrote: post #10

I remember seeing this article before. This time the reaction that surges for me is: if you want long-term archiving but don't assume redundant storage, it's not going to go well. Put your long-term archives on ZFS.

Why are you assuming they aren't assuming redundant storage? Redundant storage isn't a cure-all, there's still a chance two blocks on two disks will fail in the exact same spot.

Seems easier to increase the amount of disks and address it at a low layer than to re-engineer all layers, all file formats, for corruption.

Re: Xz format inadequate for long-term archiving (2016)

#17

Post by asveikau » Wed, Jun 05, 2019, 4:33 PM UTC

StavrosK wrote: post #15

kipari wrote: post #14

Earlier quoted context omitted.

I reckon that the chance of the same two blocks on two different disks failing between ZFS scrubs would be incredibly small.

Yes, over normal timescales. A lot can happen in a thousand years.

Thousands of years is a lot of scrubs and a lot of disk replacements, though. And a solution like ZFS, properly monitored, should help make those detections and repairs happen early, with lower odds of loss.

Although honestly in a thousand year timeframe I very much doubt humanity will preserve ZFS, gzip, tar, jpeg, PNG, ASCII, today's spoken and written languages in current form, etc. Just as written material from 1000 years ago is not very accessible to most people; with the original material you need intense study before you even know what you're looking at.

Re: Xz format inadequate for long-term archiving (2016)

#18

Post by zamalek » Wed, Jun 05, 2019, 4:43 PM UTC

LinuxBender wrote: post #3

If you first use tar to preserve xattrs/etc.. then you can use anything to compress. xz, bz2, 7z, even arj if you are feeling nostalgic. tar cvfJ ./files.tar.xz /some/dir

You've missed the point of the article entirely. A single bit-flip (which is almost guaranteed over long-term) can easily render the entire xz file corrupt.

This has nothing to do with xattrs/etc.

Re: Xz format inadequate for long-term archiving (2016)

#19

Post by wereHamster » Wed, Jun 05, 2019, 4:53 PM UTC

esaym wrote: post #9

Interestingly, since "recovery" is mentioned several times, I decided to test myself. I took a copy of a jpeg image, compressed it different times with either gzip or bzip2, then with a hexeditor modified one byte. The recovery instructions for gzip is to simply do "zcat corrupt_file.gz > corrupt_file". While for bzip2 is to use the bzip2recover command which just dumps the blocks out individually (corrupt ones and a…

Whether recovery leads to (almost) useable data depends on what byte you modify. It's entirely possible that a single corrupt byte in the compressed data leads to a single corrupt byte when uncompressed. When you are dealing with images you may not even notice that a single pixel is wrong. But it's also possible that you completely destroy the data such that the decompression algorithm can't even deal with it and has to give up.

Re: Xz format inadequate for long-term archiving (2016)

#20

Post by gmueckl » Wed, Jun 05, 2019, 4:59 PM UTC

esaym wrote: post #9

Interestingly, since "recovery" is mentioned several times, I decided to test myself. I took a copy of a jpeg image, compressed it different times with either gzip or bzip2, then with a hexeditor modified one byte. The recovery instructions for gzip is to simply do "zcat corrupt_file.gz > corrupt_file". While for bzip2 is to use the bzip2recover command which just dumps the blocks out individually (corrupt ones and a…

Your method is clearly flawed. Altering a single byte once is insufficient as a test unless you analyzed the structure of the compressed file first to see where the really important information is stored. It may well be that you just modified a verbatim string from the source data in the gzip case, but corrupted a bit of metadata about how the compressed data is structured in the bzip2 case. If you tried a different random bytes, the results might be reversed.

The proper test would be to iterate over every bit in the compressed file, flip it and try to recover. Then compute number of successful recoveries against the number of bits tested. Compression algorithms that perform similarly should gmhave similar likelyhoods that a single bit flip corrupts the entirety of the data.