Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Xz format inadequate for long-term archiving (2017)

nongnu.org

Post reply on HN
Print view View original

1–10 of 138 posts

Re: Xz format inadequate for long-term archiving (2017)

#3

Post by cpburns2009 »

It may not be a good choice for long-term data storage, but I disagree that it should not be used for data sharing or software distribution. Different use cases have different needs. If you need long-term storage, it's better to avoid lossless compression that can break after minor corruption. You should also be storing parity/ECC data (I don't recall the subtle difference). If you only need short to moderate term storage, the best compression ratio is likely optimal. Keep a spare backup just in case.

Re: Xz format inadequate for long-term archiving (2017)

#5

Post by speleo_engr »

cpburns2009 wrote: post #3

It may not be a good choice for long-term data storage, but I disagree that it should not be used for data sharing or software distribution. Different use cases have different needs. If you need long-term storage, it's better to avoid lossless compression that can break after minor corruption. You should also be storing parity/ECC data (I don't recall the subtle difference). If you only need short to moderate term st…

I've used XZ to compress tarballs of backup. XZ was useful so I could store more backups on an external hard drive. I have seen bit rot on some of these files (stored on a magnetic HDD), in the sense that the md5sum of the .tar.xz archive no longer matches when it was created. What do you suggest for creating parity/ECC in this case? I'm aware of parchive, but is that the right choice and in what configuration?

Re: Xz format inadequate for long-term archiving (2017)

#6

Post by Lionsion »

cpburns2009 wrote: post #3

It may not be a good choice for long-term data storage, but I disagree that it should not be used for data sharing or software distribution. Different use cases have different needs. If you need long-term storage, it's better to avoid lossless compression that can break after minor corruption. You should also be storing parity/ECC data (I don't recall the subtle difference). If you only need short to moderate term st…

> It may not be a good choice for long-term data storage, but I disagree that it should not be used for data sharing or software distribution. Different use cases have different needs.

I'm not so sure, using tools suitable long-term archiving by default might not be a bad practice. The thing about archiving is that it's often hard to know in advance what exactly you want to keep long-term. Using more robust formats probably won't cost much in the short term, but could pay off in the long term.

Re: Xz format inadequate for long-term archiving (2017)

#8

Post by snuxoll »

cpburns2009 wrote: post #3

It may not be a good choice for long-term data storage, but I disagree that it should not be used for data sharing or software distribution. Different use cases have different needs. If you need long-term storage, it's better to avoid lossless compression that can break after minor corruption. You should also be storing parity/ECC data (I don't recall the subtle difference). If you only need short to moderate term st…

For long-term archival I think relying on your compression software to protect data integrity is a fool's errand, protecting against bit-rot should be a function of your storage layer as long as you have control over it (in contrast to say, Usenet, where multiple providers have copies of data and you can't trust them to not lose part of it - hence the inclusion of .par files for everything under alt.binaries).

Re: Xz format inadequate for long-term archiving (2017)

#9

Post by cpburns2009 »

Lionsion wrote: post #4

What are better file formats for long term archiving? Were any of them designed specifically with that use case in mind?

There's a post on Super User that contains useful information:

"What medium should be used for long term, high volume, data storage (archival)?" https://superuser.com/q/374609/52739

It mostly focuses on the media instead of formats though.

Re: Xz format inadequate for long-term archiving (2017)

#10

Post by aidenn0 »

Lionsion wrote: post #4

What are better file formats for long term archiving? Were any of them designed specifically with that use case in mind?

TFA is on the homepage for lzip[1] which is an lzma based compressor designed for this.

You can also use xzip on top of something that can correct errors, such as par2.

1: https://www.nongnu.org/lzip/

Post reply on HN