Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Xz format inadequate for long-term archiving (2017)

nongnu.org

Post reply on HN
Print view View original

21–30 of 138 posts

Re: Xz format inadequate for long-term archiving (2017)

#21

Post by paulmd »

cpburns2009 wrote: post #9
Lionsion wrote: post #4

What are better file formats for long term archiving? Were any of them designed specifically with that use case in mind?

There's a post on Super User that contains useful information: "What medium should be used for long term, high volume, data storage (archival)?" https://superuser.com/q/374609/52739 It mostly focuses on the media instead of formats though.

It all depends on what your definition of "high-volume" is, and just how "archival" your access patterns really are.

Amazon Glacier runs on BDXL disc libraries (like a tape library). There's nothing truly expensive about producing BDXL media, there just isn't enough volume in the consumer market to make it worthwhile. If you contract directly with suppliers for a few million discs at a time, that's not an issue (you did say high-volume, right?).

https://storagemojo.com/2014/04/25/amazons-glacier-secret-bd...

For medium-scale users, tape libraries are still the way to go. You can have petabytes of near-line storage in a rack. Storage conditions are not really a concern in a datacenter, which is where they should live.

(CERN has about 200 petabytes of tapes for their long-term storage.)

https://home.cern/about/updates/2017/07/cern-data-centre-pas...

If you mean "high-volume for a small business", probably also tapes, or BD discs with 20% parity encoding to guard against bitrot.

Small users should also consider dumping it in Glacier as a fallback - make it Amazon's problem. If you have a significant stream of data it'll get expensive over time, but if it's business-critical data then you don't really have a choice, do you?

Re: Xz format inadequate for long-term archiving (2017)

#22

Post by kazinator »

> The xz format lacks a version number field. The only reliable way of knowing if a given version of a xz decompressor can decompress a given file is by trial and error.

Wow ... that is inexcusably idiotic. Whoever designed that shouldn't be programming. Out of professional disdain, I pledge never to use this garbage.

Re: Xz format inadequate for long-term archiving (2017)

#24

Post by Lionsion »

hzhou321 wrote: post #16
Lionsion wrote: post #4

What are better file formats for long term archiving? Were any of them designed specifically with that use case in mind?

According to the article, xz minus extensibility.

That's not what I got from it. Xz has other problems such as:

> According to [Koopman] (p. 50), one of the "Seven Deadly Sins" (i.e., bad ideas) of CRC and checksum use is failing to protect a message length field. This causes vulnerabilities due to framing errors. Note that the effects of a framing error in a data stream are more serious than what Figure 1 suggests. Not only data at a random position are interpreted as the CRC. Whatever data that follow the bogus CRC will be interpreted as the beginning of the following field, preventing the successful decoding of any remaining data in the stream.

> Except the 'Backward Size' field in the stream footer, none of the many length fields in the xz format is protected by a check sequence of any kind. Not even a parity bit. All of them suffer from the framing vulnerability illustrated in the picture above.

Re: Xz format inadequate for long-term archiving (2017)

#25

Post by zokier »

cpburns2009 wrote: post #3

It may not be a good choice for long-term data storage, but I disagree that it should not be used for data sharing or software distribution. Different use cases have different needs. If you need long-term storage, it's better to avoid lossless compression that can break after minor corruption. You should also be storing parity/ECC data (I don't recall the subtle difference). If you only need short to moderate term st…

Of course there are million variables here, but for compressible data arguably compression+ecc is more robust against damage than uncompressed data. The rationale being that with compression you can afford to use more/bigger ecc

Re: Xz format inadequate for long-term archiving (2017)

#26

Post by wyldfire »

arundelo wrote: post #17

I upvoted this because it seems to make some good points and I think the topic is interesting and important, but I can't understand why the "Then, why some free software projects use xz?" section does not mention xz's main selling point of being better than other commonly used alternatives at compressing things to smaller sizes. https://www.rootusers.com/gzip-vs-bzip2-vs-xz-performance-co...

> compressing things to smaller sizes.

...relative to ... ? Is it better than lzip? lzip sounds like it would also use LZMA-based compression, right? This [1] sounds like an interesting and more detailed/up-to-date comparison. Also by the same author BTW.

[1] https://www.nongnu.org/lzip/lzip_benchmark.html#xz

Re: Xz format inadequate for long-term archiving (2017)

#27

Post by zokier »

cpburns2009 wrote: post #9
Lionsion wrote: post #4

What are better file formats for long term archiving? Were any of them designed specifically with that use case in mind?

There's a post on Super User that contains useful information: "What medium should be used for long term, high volume, data storage (archival)?" https://superuser.com/q/374609/52739 It mostly focuses on the media instead of formats though.

Personally I think the premise of the question is poor. Attempting to build monolithic long term (100+ years) cold storage of significant amount of data is a folly, instead the only reasonable approach is to do it in smaller parts (maybe 10-20 years) and plan for migrations.

Re: Xz format inadequate for long-term archiving (2017)

#29

Post by tedunangst »

Are these concerns, about error recovery, outdated? If I want to recover a corrupted file, I find another copy. I don't fiddle with the internal length field to fix framing issues. Certainly, if I want to detect corruption, I use a sha256 of the entire file. If that fails, I don't waste time trying to find the bad bit.

To add to that, if you need parity to recover from errors, you need to calculate how much based on your storage medium durability and projected life span. It's not the file format's concern. The xz crc should be irrelevant.

Re: Xz format inadequate for long-term archiving (2017)

#30

Post by zzzcpan »

kazinator wrote: post #22

> The xz format lacks a version number field. The only reliable way of knowing if a given version of a xz decompressor can decompress a given file is by trial and error. Wow ... that is inexcusably idiotic. Whoever designed that shouldn't be programming. Out of professional disdain, I pledge never to use this garbage.

Welcome to the world of software I guess. Non idiotic things are rare here.
Post reply on HN