Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Xz format inadequate for long-term archiving (2017)

nongnu.org

Post reply on HN
Print view View original

71–80 of 138 posts

Re: Xz format inadequate for long-term archiving (2017)

#71

Post by jjuhl »

tedunangst wrote: post #29

Are these concerns, about error recovery, outdated? If I want to recover a corrupted file, I find another copy. I don't fiddle with the internal length field to fix framing issues. Certainly, if I want to detect corruption, I use a sha256 of the entire file. If that fails, I don't waste time trying to find the bad bit. To add to that, if you need parity to recover from errors, you need to calculate how much based on…

If you are archiving it may be for long times (think "museums", "data vaults" etc). Finding another copy 200 years later may be difficult.

Re: Xz format inadequate for long-term archiving (2017)

#72

Post by joombaga »

rikkus wrote: post #62
jlgaddis wrote: post #45

Earlier quoted context omitted.

> So you've archived two or more copies of each file You haven't? It took me just one minor "data loss incident" ~20 years ago to very quickly convince me to become a lifetime member of the "backup all the things to a few different locations" club. > That means you're use at least twice as much space (and if you're keeping the original as well, more than twice). "Storage is cheap."

Storage is cheap indeed, though it takes some effort to make it cheap. 99% of the digital data I'm keeping for the long term is family photos and videos. All my photos go to Dropbox (easy copy-from-device and access anywhere) and are then backed up to multiple locations by CrashPlan. It'll be a while yet, but in the next few years I'll be hitting the 1TB Dropbox limit. I'm hoping that Dropbox make a >1TB 'consumer' p…

Are you using the small business version of CrashPlan? I was using CP too until they discontinued their B2C.

Re: Xz format inadequate for long-term archiving (2017)

#73

Post by tedunangst »

pmoriarty wrote: post #42
tedunangst wrote: post #29

Are these concerns, about error recovery, outdated? If I want to recover a corrupted file, I find another copy. I don't fiddle with the internal length field to fix framing issues. Certainly, if I want to detect corruption, I use a sha256 of the entire file. If that fails, I don't waste time trying to find the bad bit. To add to that, if you need parity to recover from errors, you need to calculate how much based on…

"If I want to recover a corrupted file, I find another copy." So you've archived two or more copies of each file? That means you're use at least twice as much space (and if you're keeping the original as well, more than twice). For the likely corruption of the occasional single bit flip here and there, you could do a lot better by using something like par2 and/or dvdisaster (depending on what media you're archiving t…

If you're using par2, I'd say that's closer to recovering a second copy than trying to extract meaningful data from a corrupted file. (The internal structure of the format is irrelevant, thus concerns about it are outdated.)

Re: Xz format inadequate for long-term archiving (2017)

#74

Post by foepys »

jwilliams wrote: post #47

I sent a reasonable amount of data to Cloud Storage. It varies a lot. Usually ~10GB/day, but it can be up to 1TB/day regularly. xz can be amazing . It can also bite you. I've had payloads that compress to 0.16 with gzip then compress to 0.016 with xz. Hurray! Then I've had payloads where xz compression is par, or worse. However, with "best or extreme" compression, xz can peg your CPU for much longer. gzip and bzip2 w…

If you can compress data this much, you seem to have a lot of repetitive data. Have you tried using compression algorithms that support custom dictionaries? ZSTD and DEFLATE support those and can maybe help with compression ratio as well as speed.

Re: Xz format inadequate for long-term archiving (2017)

#75

Post by pmoriarty »

tedunangst wrote: post #73
pmoriarty wrote: post #42

Earlier quoted context omitted.

"If I want to recover a corrupted file, I find another copy." So you've archived two or more copies of each file? That means you're use at least twice as much space (and if you're keeping the original as well, more than twice). For the likely corruption of the occasional single bit flip here and there, you could do a lot better by using something like par2 and/or dvdisaster (depending on what media you're archiving t…

If you're using par2, I'd say that's closer to recovering a second copy than trying to extract meaningful data from a corrupted file. (The internal structure of the format is irrelevant, thus concerns about it are outdated.)

Even when using par2, the fact that xz files contain no version number for the format is still troubling.

Re: Xz format inadequate for long-term archiving (2017)

#76

Post by jwilliams »

londons_explore wrote: post #52
jwilliams wrote: post #47

I sent a reasonable amount of data to Cloud Storage. It varies a lot. Usually ~10GB/day, but it can be up to 1TB/day regularly. xz can be amazing . It can also bite you. I've had payloads that compress to 0.16 with gzip then compress to 0.016 with xz. Hurray! Then I've had payloads where xz compression is par, or worse. However, with "best or extreme" compression, xz can peg your CPU for much longer. gzip and bzip2 w…

If you get compression ratios that good, you should consider if your application might be doing something stupid like storing the same data thousands of times inside it's data file. If you store enough of the same type of data, invest in redesigning the application. There's a reason we all use jpegs over zipped bitmaps...

> There's a reason we all use jpegs over zipped bitmaps...

It's because it's an appropriate compression - just like xz can be? Not sure what you're actually suggesting here.

Re: Xz format inadequate for long-term archiving (2017)

#77

Post by outworlder »

pmoriarty wrote: post #42
tedunangst wrote: post #29

Are these concerns, about error recovery, outdated? If I want to recover a corrupted file, I find another copy. I don't fiddle with the internal length field to fix framing issues. Certainly, if I want to detect corruption, I use a sha256 of the entire file. If that fails, I don't waste time trying to find the bad bit. To add to that, if you need parity to recover from errors, you need to calculate how much based on…

"If I want to recover a corrupted file, I find another copy." So you've archived two or more copies of each file? That means you're use at least twice as much space (and if you're keeping the original as well, more than twice). For the likely corruption of the occasional single bit flip here and there, you could do a lot better by using something like par2 and/or dvdisaster (depending on what media you're archiving t…

Yes.

If your data is not in three different places it might as well not exist.

Re: Xz format inadequate for long-term archiving (2017)

#78

Post by freeone3000 »

chatmasta wrote: post #66
rspeer wrote: post #61

Earlier quoted context omitted.

HTML is pretty repetitive, but if you want to archive HTML data, you don't get to redefine what HTML is. Compression is useful.

This is what the WARC [0] file format (and/or gzip) is for. [0] https://en.m.wikipedia.org/wiki/Web_ARChive

and/or xz? because xz gives better compression than gzip or warc?

Re: Xz format inadequate for long-term archiving (2017)

#79

Post by comboy »

jlgaddis wrote: post #68
pmoriarty wrote: post #51

Earlier quoted context omitted.

"You haven't?" No I don't, because it's a waste of space and money. Using par2 and/or dvdisaster I can archive a lot more files on to the same archival media and still get enough redundancy to feel secure. "Storage is cheap." Cheap is relative. Are you really buying an extra 2 TB's of storage to archive 1 TB of data, because that's what you'd need to do to archive 2 copies of each file. That's a huge waste of space a…

> Are you really buying an extra 2 TB's of storage to archive 1 TB of data ... $ sudo zpool get size zdata NAME PROPERTY VALUE SOURCE zdata size 21.8T - Yep. It's fine that you "feel secure" with your current backup regimen -- and I certainly hope you never lose any important data. After losing data once, though, I promised myself I'd do my best to make sure that it never happened again. The "primary copy" of all my…

> Some of us just want a little bit more reassurance than others

Which gets back to the original point. If you use format that can be more easily recovered, then having the same amount of copies, you're data is more secure.

You've probably been downvoted because it's perceived as showing off, but it is a nice setup.

I've also spend more time than I'm willing to admit with planning researching configuring and maintaining different backup strategies, and just wanted to say that I regret some of that. It's easy to become data hoarder and it's easy to spend more time on preserving it than it is actually worth. I mean, think about how much of this data is worth to people other than you, i.e. what happens to it when you die. Life's short and there are so many things that are more exciting than backups.

Don't get me wrong though. Backups are important.Just know how much exactly are they important to you.

Post reply on HN