Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Xz format inadequate for long-term archiving (2016)

lzip.nongnu.org

Post reply on HN
Print view View original

41–50 of 60 posts

Re: Xz format inadequate for long-term archiving (2016)

#41

Post by dual_basis »

esaym wrote: post #21
gmueckl wrote: post #20

Earlier quoted context omitted.

Your method is clearly flawed. Altering a single byte once is insufficient as a test unless you analyzed the structure of the compressed file first to see where the really important information is stored. It may well be that you just modified a verbatim string from the source data in the gzip case, but corrupted a bit of metadata about how the compressed data is structured in the bzip2 case. If you tried a different…

I thought about that as well. I tried it three different times all with the same results.

Three? Well then, case closed!

Re: Xz format inadequate for long-term archiving (2016)

#42

Post by imiric »

zamalek wrote: post #18
LinuxBender wrote: post #3

If you first use tar to preserve xattrs/etc.. then you can use anything to compress. xz, bz2, 7z, even arj if you are feeling nostalgic. tar cvfJ ./files.tar.xz /some/dir

You've missed the point of the article entirely. A single bit-flip (which is almost guaranteed over long-term ) can easily render the entire xz file corrupt. This has nothing to do with xattrs/etc.

I've been using tar+pixz+par2 for backups for a while now, but this article still worries me.

What can I replace pixz with that compresses as well and keeps the indexing functionality? I'd like to avoid zstd because Facebook.

Re: Xz format inadequate for long-term archiving (2016)

#43

Post by ltbarcly3 »

matt-attack wrote: post #39
ltbarcly3 wrote: post #35

Earlier quoted context omitted.

Why would they need to recreate it solely based on 'documentation'? It is open source, the source code is the documentation. It seems just as likely that the source would survive as it is likely that some complete technical documentation would survive. Maybe they wouldn't be able to compile it (probably they would be able to compile it, I don't see why they wouldn't have some kind of computer emulator available), but…

You're exhibiting shortsightedness when it comes to "source". If I give you some RPG [1] or maybe some ALGO 58 [2] source code are you going to just compile and run it no problem? How about some FLOW-MATIC [3]? Point being that computer languages come and go. [1] https://en.wikipedia.org/wiki/IBM_RPG [2] https://en.wikipedia.org/wiki/ALGOL_58 [3] https://en.wikipedia.org/wiki/FLOW-MATIC

Yes, programming languages come and go, but I don't see how that matters. Some future historian will either have access to a working copy of xz or they will not. If they don't, and they want to implement it, having a copy of the source code is far better than anything else you could give them. Sure, future programming languages will be quite different, but humans will certainly be able to read and understand C code. If humanity has forgotten how to read C code (and lost all knowledge of it), how are they going to read this documentation you seem to prefer? Human languages come and go also..

https://en.wikipedia.org/wiki/Egyptian_hieroglyphs

https://en.wikipedia.org/wiki/Judaeo-Aragonese

https://en.wikipedia.org/wiki/Latin

Any argument you can make about historians being able to recover dead languages you can make the exact same argument for their ability to recover dead computer languages, and there is no better or more accurate specification than the actual code.

So let me add to my recommendation, in addition to a copy of the xz source code, include a plain text copy of any 'how to program in C' book, or just the wikipedia page for the C language. That is more than enough for them to construct a program that can decompress xz files, once they relearn how to read whatever long dead language the book is written in (Ancient Pre-Cataclysm Earth English for example).

Re: Xz format inadequate for long-term archiving (2016)

#44

Post by zaphirplane »

cmurf wrote: post #37
kipari wrote: post #14

Earlier quoted context omitted.

I reckon that the chance of the same two blocks on two different disks failing between ZFS scrubs would be incredibly small.

It is incredibly small if you don't consider either drive failing. But if one drive fails, it happens with some regularity that a sector on the good drive is bad. In actuality, only one sector is bad, but in effect the dead drive means its mirror is also bad. This comes up on the linux raid list with some frequency whenever there are drive failures with raid56, and the subsequently the raid trips over a single bad se…

Are you saying that a failure happens which isn’t detected and when the 2nd failure occurs we notice because the data is inaccessible?

Which raid s/w does this ?

Re: Xz format inadequate for long-term archiving (2016)

#45

Post by rincebrain »

kipari wrote: post #14
StavrosK wrote: post #12

Earlier quoted context omitted.

Why are you assuming they aren't assuming redundant storage? Redundant storage isn't a cure-all, there's still a chance two blocks on two disks will fail in the exact same spot.

I reckon that the chance of the same two blocks on two different disks failing between ZFS scrubs would be incredibly small.

Assuming the corruption is independent, potentially, but A) even unlikely events are likely to happen for large enough N, and much more importantly, B) as another poster described, if you don't regularly check the integrity, and you have single-disk redundancy, losing a whole disk can likely result in you discovering a block that got mangled some time ago, too late to do anything about it.

There are a number of cases where failures might not be independent, though.

What if, say, you're using multiple drives of the same model, which have a firmware bug causing them to sometimes mangle data on the Nth sector?

What if you're using multiple drives from the same manufacturing batch which have a flaw leading to certain regions being more likely to fail than others?

What if you're using some battery-backed write cache under ZFS (from a HW RAID card or something more exotic), and it helpfully writes out garbage to the same sector on two disks?

What if you have a certain manufacturer's hard drives that lie about flushing their write cache successfully to disk if you issue a SMART request to them between when they put data in cache and when it actually gets to disk, so polling those two disks when they both just got a write results in data loss?

(The last of these is a real firmware bug I ran into - I was running a testbed of a bunch of raidz3 vdevs, and spent some time isolating when zpool scrub kept making the error counters increase even though it had corrected them all...thanks, Samsung HD204UI drives.)

Re: Xz format inadequate for long-term archiving (2016)

#46

Post by fao_ »

dual_basis wrote: post #41
esaym wrote: post #21

Earlier quoted context omitted.

I thought about that as well. I tried it three different times all with the same results.

Three? Well then, case closed!

Did the poster imply that their test was the be-all and end-all of error tolerance in common-use compressions systems? No. Then why did you assume that they did say that, and then write such a useless comment

Re: Xz format inadequate for long-term archiving (2016)

#47

Post by fao_ »

ltbarcly3 wrote: post #43
matt-attack wrote: post #39

Earlier quoted context omitted.

You're exhibiting shortsightedness when it comes to "source". If I give you some RPG [1] or maybe some ALGO 58 [2] source code are you going to just compile and run it no problem? How about some FLOW-MATIC [3]? Point being that computer languages come and go. [1] https://en.wikipedia.org/wiki/IBM_RPG [2] https://en.wikipedia.org/wiki/ALGOL_58 [3] https://en.wikipedia.org/wiki/FLOW-MATIC

Yes, programming languages come and go, but I don't see how that matters. Some future historian will either have access to a working copy of xz or they will not. If they don't, and they want to implement it, having a copy of the source code is far better than anything else you could give them. Sure, future programming languages will be quite different, but humans will certainly be able to read and understand C code.…

> If humanity has forgotten how to read C code (and lost all knowledge of it), how are they going to read this documentation you seem to prefer?

Sure but are they going to remember something like, weird precedence rules (See: &), undefined behaviour, etc. Just because they want to reimplement a specific, small, program does not mean they want to relearn several languages. What you're saying could easily blow up from 'how to code C' to 'reading the GCC / Clang compiler source code to figure out how a specific UB was implemented, which the program in this specific case falls into', which I'm sure nobody wants to spend their weekend doing, implementing something like `xz` could simply be a midpoint in their destination, they don't want to spend weeks digging up COBOL. Have at least some consideration for the human element, jeez.

Documentation, specifically _mathematical_ documentation, is more fault tolerant than either psuedocode or actual code.

At any other time, I would agree with you, but where archivism is concerned, I do not.

Re: Xz format inadequate for long-term archiving (2016)

#48

Post by cmurf »

zaphirplane wrote: post #44
cmurf wrote: post #37

Earlier quoted context omitted.

It is incredibly small if you don't consider either drive failing. But if one drive fails, it happens with some regularity that a sector on the good drive is bad. In actuality, only one sector is bad, but in effect the dead drive means its mirror is also bad. This comes up on the linux raid list with some frequency whenever there are drive failures with raid56, and the subsequently the raid trips over a single bad se…

Are you saying that a failure happens which isn’t detected and when the 2nd failure occurs we notice because the data is inaccessible? Which raid s/w does this ?

Correct. All that depend on the SCSI block layer, which includes libata and thus common consumer SATA drives. A NAS or better drive will come out of the box with short error time outs, typically 70 deciseconds, and quickly issue a read error with the LBA of the offending bad sector, and the RAID can then know to obtain a copy or reconstruct from parity, write the good data to the bad sector thus fixing it. Either the write works, or if it fails the drive firmware is responsible for remapping that LBA to a reserve physical sector.

In the case where the drive error timeout is longer than the SCSI block layer, it just results in a link reset. The actual problem with the drive is obscured by the reset, including the bad sector, so it never gets repaired.

Btrfs, mdadm, lvm are affected and I'm pretty sure ZFS on Linux as well assuming they haven't totally reimplemented their own block layer outside of the SCSI subsystem.

It's a super irritating problem, the kernel developers know all about it, but thus far it's considered something distributions should change for the use cases that need it. And what that means so far is distros don't change it and users using consumer drives with high error recovery times, get bitten.

https://raid.wiki.kernel.org/index.php/Timeout_Mismatch

Re: Xz format inadequate for long-term archiving (2016)

#49

Post by matt-attack »

xoa wrote: post #29
matt-attack wrote: post #23

Earlier quoted context omitted.

I think with archiving it’s more than that. Sure you can guarantee that the actual tool you just compressed with can restore the original perfectly. But with long term digital archiving I think you need the assurance that the “spec” called “xz” could be perfectly reimplemented by an expert in the future. Based solely on documentation. And on a platform that doesn’t exist today. That is, you must assume the original e…

I don't disagree with that at all! I did try to say (maybe unclearly) that having simpler more foolproof and failsafe layers at every level seems absolutely worth pursuing anyway, where possible. But I also wonder whether some of the common wisdom is from an age that is obsolete for many scenarios? Ie., in the 80s and 90s and even early 00s there was a lot more churn, practices were less standardized, computing time…

> many long term archives should simply including everything necessary to deal with them in the present day as a container or VM image

Unfortunately, any such image would presume you have access to the hardware, or it has low-level instruction sets/processor design baked in. Think how many PDP-11's are around today. And in terms of an archive it's only been 50 years since the PDP-11 was invented. That's a blink of an eye in terms of archival standards.

Re: Xz format inadequate for long-term archiving (2016)

#50

Post by theamk »

matt-attack wrote: post #49
xoa wrote: post #29

Earlier quoted context omitted.

I don't disagree with that at all! I did try to say (maybe unclearly) that having simpler more foolproof and failsafe layers at every level seems absolutely worth pursuing anyway, where possible. But I also wonder whether some of the common wisdom is from an age that is obsolete for many scenarios? Ie., in the 80s and 90s and even early 00s there was a lot more churn, practices were less standardized, computing time…

> many long term archives should simply including everything necessary to deal with them in the present day as a container or VM image Unfortunately, any such image would presume you have access to the hardware, or it has low-level instruction sets/processor design baked in. Think how many PDP-11's are around today. And in terms of an archive it's only been 50 years since the PDP-11 was invented. That's a blink of an…

> Think how many PDP-11's are around today.

Why does it matter how many physical machines are alive? There are tons of emulators around. There is even one in Javascript, with an ability to load disk images as well.

The hard part is hardware - the drives go bad, the computers fail. But disks grow, and it is getting simpler and cheaper to store lots of data. As long as you keep copying the files to modern media every 10 years or so, you should no longer have anybdata loss.

(The only exception is proprietary data formats which cannot be opened except by original program which cannot be run in VM easily. Those should be avoided at all costs)

Post reply on HN