Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Xz format inadequate for long-term archiving (2017)

nongnu.org

Post reply on HN
Print view View original

11–20 of 138 posts

Re: Xz format inadequate for long-term archiving (2017)

#11

Post by aidenn0 »

saghm wrote: post #7
nurettin wrote: post #2

Too bad for arch https://www.archlinux.org/news/switching-to-xz-compression-f...

Is this really an issue for this use case? My naive take is that since Arch updates packages so often, "long-term storage" doesn't come up that much in practice.

It's zero issue since the packages are updated regularly and hashed before installing.

Re: Xz format inadequate for long-term archiving (2017)

#12

Post by Lionsion »

cpburns2009 wrote: post #9
Lionsion wrote: post #4

What are better file formats for long term archiving? Were any of them designed specifically with that use case in mind?

There's a post on Super User that contains useful information: "What medium should be used for long term, high volume, data storage (archival)?" https://superuser.com/q/374609/52739 It mostly focuses on the media instead of formats though.

Thanks, I'll take a look. Though I think I have the media question answered, and I settled on M-DISC for personal stuff (https://en.wikipedia.org/wiki/M-DISC). It only has special requirements for writing, reading can be done on standard drives.

Re: Xz format inadequate for long-term archiving (2017)

#13

Post by doubledad222 »

Thank you for sharing this. I am in charge of archiving the family files - pictures, video, art projects, email. I want it available through the aging of standards and protected against the bitrot of aging hard drives. I'll be converting any xz archives I get into a better format.

Re: Xz format inadequate for long-term archiving (2017)

#15

Post by yorwba »

cpburns2009 wrote: post #3

It may not be a good choice for long-term data storage, but I disagree that it should not be used for data sharing or software distribution. Different use cases have different needs. If you need long-term storage, it's better to avoid lossless compression that can break after minor corruption. You should also be storing parity/ECC data (I don't recall the subtle difference). If you only need short to moderate term st…

I have occasionally had downloaded tarballs that were truncated by network failure. It's nice to be able to get a meaningful error when decompression fails, instead of silently decompressing only part of the data. So built-in integrity checks are also desirable for short-term distribution.

Re: Xz format inadequate for long-term archiving (2017)

#17

Post by arundelo »

I upvoted this because it seems to make some good points and I think the topic is interesting and important, but I can't understand why the "Then, why some free software projects use xz?" section does not mention xz's main selling point of being better than other commonly used alternatives at compressing things to smaller sizes.

https://www.rootusers.com/gzip-vs-bzip2-vs-xz-performance-co...

Re: Xz format inadequate for long-term archiving (2017)

#18

Post by cpburns2009 »

speleo_engr wrote: post #5
cpburns2009 wrote: post #3

It may not be a good choice for long-term data storage, but I disagree that it should not be used for data sharing or software distribution. Different use cases have different needs. If you need long-term storage, it's better to avoid lossless compression that can break after minor corruption. You should also be storing parity/ECC data (I don't recall the subtle difference). If you only need short to moderate term st…

I've used XZ to compress tarballs of backup. XZ was useful so I could store more backups on an external hard drive. I have seen bit rot on some of these files (stored on a magnetic HDD), in the sense that the md5sum of the .tar.xz archive no longer matches when it was created. What do you suggest for creating parity/ECC in this case? I'm aware of parchive, but is that the right choice and in what configuration?

Keep in mind I'm not an archival expert so you should do your own research. That being said, currently I'm using pyFileFixity [1] to generate the hashes and ECC data for my personal backups. I write them to M-Disc Blu-rays using Dvdisaster [2] which can also write additional ECC data. After a lot of googling and reading this useful Super User question [3], and this extensive answer [4] I settled on this setup. I must admit that I am guilty of storing images as JPGs and compressing most most of my files in ZIPs for convenience.

[1]: https://github.com/lrq3000/pyFileFixity

[2]: http://dvdisaster.net/en/index.html

[3]: https://superuser.com/q/374609/52739

[4]: https://superuser.com/a/873260/52739

Re: Xz format inadequate for long-term archiving (2017)

#19

Post by cpburns2009 »

Lionsion wrote: post #12
cpburns2009 wrote: post #9

Earlier quoted context omitted.

There's a post on Super User that contains useful information: "What medium should be used for long term, high volume, data storage (archival)?" https://superuser.com/q/374609/52739 It mostly focuses on the media instead of formats though.

Thanks, I'll take a look. Though I think I have the media question answered, and I settled on M-DISC for personal stuff ( https://en.wikipedia.org/wiki/M-DISC ). It only has special requirements for writing, reading can be done on standard drives.

I went with M-Disc too and an LG Blu-ray burner. I think you only need a special burner if you're using the DVDs. I want to say most Blu-ray burners work.

Re: Xz format inadequate for long-term archiving (2017)

#20

Post by dv_dt »

snuxoll wrote: post #8
cpburns2009 wrote: post #3

It may not be a good choice for long-term data storage, but I disagree that it should not be used for data sharing or software distribution. Different use cases have different needs. If you need long-term storage, it's better to avoid lossless compression that can break after minor corruption. You should also be storing parity/ECC data (I don't recall the subtle difference). If you only need short to moderate term st…

For long-term archival I think relying on your compression software to protect data integrity is a fool's errand, protecting against bit-rot should be a function of your storage layer as long as you have control over it (in contrast to say, Usenet, where multiple providers have copies of data and you can't trust them to not lose part of it - hence the inclusion of .par files for everything under alt.binaries).

I keep seeing recommendations for par/par2 but it seems like as software, the project isn't actively maintained? As an aside, that makes me think of dead languages and the use of latin for scientific names because it isn't changing anymore... but do you want that out of archival formats and software?
Post reply on HN