Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Xz format inadequate for long-term archiving (2017)

nongnu.org

Post reply on HN
Print view View original

61–70 of 138 posts

Re: Xz format inadequate for long-term archiving (2017)

#61

Post by rspeer »

londons_explore wrote: post #52
jwilliams wrote: post #47

I sent a reasonable amount of data to Cloud Storage. It varies a lot. Usually ~10GB/day, but it can be up to 1TB/day regularly. xz can be amazing . It can also bite you. I've had payloads that compress to 0.16 with gzip then compress to 0.016 with xz. Hurray! Then I've had payloads where xz compression is par, or worse. However, with "best or extreme" compression, xz can peg your CPU for much longer. gzip and bzip2 w…

If you get compression ratios that good, you should consider if your application might be doing something stupid like storing the same data thousands of times inside it's data file. If you store enough of the same type of data, invest in redesigning the application. There's a reason we all use jpegs over zipped bitmaps...

HTML is pretty repetitive, but if you want to archive HTML data, you don't get to redefine what HTML is. Compression is useful.

Re: Xz format inadequate for long-term archiving (2017)

#62

Post by rikkus »

jlgaddis wrote: post #45
pmoriarty wrote: post #42

Earlier quoted context omitted.

"If I want to recover a corrupted file, I find another copy." So you've archived two or more copies of each file? That means you're use at least twice as much space (and if you're keeping the original as well, more than twice). For the likely corruption of the occasional single bit flip here and there, you could do a lot better by using something like par2 and/or dvdisaster (depending on what media you're archiving t…

> So you've archived two or more copies of each file You haven't? It took me just one minor "data loss incident" ~20 years ago to very quickly convince me to become a lifetime member of the "backup all the things to a few different locations" club. > That means you're use at least twice as much space (and if you're keeping the original as well, more than twice). "Storage is cheap."

Storage is cheap indeed, though it takes some effort to make it cheap.

99% of the digital data I'm keeping for the long term is family photos and videos. All my photos go to Dropbox (easy copy-from-device and access anywhere) and are then backed up to multiple locations by CrashPlan.

It'll be a while yet, but in the next few years I'll be hitting the 1TB Dropbox limit. I'm hoping that Dropbox make a >1TB 'consumer' plan in the next couple of years. There's no way I'm assuming my backups are fine, deleting from Dropbox to make space, then finding out in a few years that some set of photos is missing.

I also sync up to Google Drive - but again, there's a 1TB limit (or a large cost).

In the future, I might have to create a new Dropbox account and keep the old one running. Storage might be cheap, but keeping it cheap is tricky.

Re: Xz format inadequate for long-term archiving (2017)

#63

Post by OldSchoolJohnny »

pmoriarty wrote: post #51
jlgaddis wrote: post #45

Earlier quoted context omitted.

> So you've archived two or more copies of each file You haven't? It took me just one minor "data loss incident" ~20 years ago to very quickly convince me to become a lifetime member of the "backup all the things to a few different locations" club. > That means you're use at least twice as much space (and if you're keeping the original as well, more than twice). "Storage is cheap."

"You haven't?" No I don't, because it's a waste of space and money. Using par2 and/or dvdisaster I can archive a lot more files on to the same archival media and still get enough redundancy to feel secure. "Storage is cheap." Cheap is relative. Are you really buying an extra 2 TB's of storage to archive 1 TB of data, because that's what you'd need to do to archive 2 copies of each file. That's a huge waste of space a…

If you have any responsibility for data protection I urge you to read literally anything on disaster recovery procedures.

Re: Xz format inadequate for long-term archiving (2017)

#64

Post by derefr »

wyldfire wrote: post #26
arundelo wrote: post #17

I upvoted this because it seems to make some good points and I think the topic is interesting and important, but I can't understand why the "Then, why some free software projects use xz?" section does not mention xz's main selling point of being better than other commonly used alternatives at compressing things to smaller sizes. https://www.rootusers.com/gzip-vs-bzip2-vs-xz-performance-co...

> compressing things to smaller sizes. ...relative to ... ? Is it better than lzip? lzip sounds like it would also use LZMA-based compression, right? This [1] sounds like an interesting and more detailed/up-to-date comparison. Also by the same author BTW. [1] https://www.nongnu.org/lzip/lzip_benchmark.html#xz

Relative to the compression formats people were aware of at the time (which didn't include lzip.)

People began using xz because mostly because they (e.g. distro maintainers like Debian) had started seeing 7z files floating around, thought they were cool, and so wanted a format that did what 7z did but was an open standard rather than being dictated by some company. xz was that format, so they leapt on it.

As it turns out, lzip had already been around for a year (though I'm not sure in what state of usability) before the xz project was started, but the people who created xz weren't looking for something that compressed better, they were looking for something that compressed better like 7z, and xz is that.

(Meanwhile, what 7z/xz is actually better at, AFAIK, is long-range identical-run deduplication; this is what makes it the tool of choice in the video-game archival community for making archives of every variation of a ROM file. Stick 100 slight variations of a 5MB file together into one .7z (or .tar.xz) file, and they'll compress down to roughly 1.2x the size of a single variant of the file.)

Re: Xz format inadequate for long-term archiving (2017)

#65

Post by leni536 »

I fail to see why integrity checking is the file format's responsibility. Is this historical? Like when you just dd a tar file directly onto a tape and there is no filesystem? Anyway seems like it should be handled by the filesystem and network layers.

I can understand the concerns about versioning and fragmented extension implementations though.

Re: Xz format inadequate for long-term archiving (2017)

#66

Post by chatmasta »

rspeer wrote: post #61
londons_explore wrote: post #52

Earlier quoted context omitted.

If you get compression ratios that good, you should consider if your application might be doing something stupid like storing the same data thousands of times inside it's data file. If you store enough of the same type of data, invest in redesigning the application. There's a reason we all use jpegs over zipped bitmaps...

HTML is pretty repetitive, but if you want to archive HTML data, you don't get to redefine what HTML is. Compression is useful.

This is what the WARC [0] file format (and/or gzip) is for.

[0] https://en.m.wikipedia.org/wiki/Web_ARChive

Re: Xz format inadequate for long-term archiving (2017)

#67

Post by planteen »

Filligree wrote: post #35
speleo_engr wrote: post #5

Earlier quoted context omitted.

I've used XZ to compress tarballs of backup. XZ was useful so I could store more backups on an external hard drive. I have seen bit rot on some of these files (stored on a magnetic HDD), in the sense that the md5sum of the .tar.xz archive no longer matches when it was created. What do you suggest for creating parity/ECC in this case? I'm aware of parchive, but is that the right choice and in what configuration?

When you're transferring files and need to cope with corrupted/missing chunks, you should use a parity scheme. Others have mentioned that; it's common for, for example, Usenet. If you can't control the underlying storage, then ditto. Keeping and maintaining explicit parity chunks is somewhat inconvenient, but it works. But if you just want to avoid bitrot of your own files, sitting on your own HDD, I'd recommend usin…

If I am using a single HDD, can ZFS still add parity data? That's neat if it can. I assumed parity with ZFS was for something like RAID6 where there are multiple HDDs in a set.

Do any other file systems other than ZFS support adding parity in a single HDD config? Last I checked getting ZFS in Linux required lots of side band steps due to licensing issues.

Re: Xz format inadequate for long-term archiving (2017)

#68

Post by jlgaddis »

pmoriarty wrote: post #51
jlgaddis wrote: post #45

Earlier quoted context omitted.

> So you've archived two or more copies of each file You haven't? It took me just one minor "data loss incident" ~20 years ago to very quickly convince me to become a lifetime member of the "backup all the things to a few different locations" club. > That means you're use at least twice as much space (and if you're keeping the original as well, more than twice). "Storage is cheap."

"You haven't?" No I don't, because it's a waste of space and money. Using par2 and/or dvdisaster I can archive a lot more files on to the same archival media and still get enough redundancy to feel secure. "Storage is cheap." Cheap is relative. Are you really buying an extra 2 TB's of storage to archive 1 TB of data, because that's what you'd need to do to archive 2 copies of each file. That's a huge waste of space a…

> Are you really buying an extra 2 TB's of storage to archive 1 TB of data ...

  $ sudo zpool get size zdata
  NAME   PROPERTY  VALUE  SOURCE
  zdata  size      21.8T  -
Yep.

It's fine that you "feel secure" with your current backup regimen -- and I certainly hope you never lose any important data.

After losing data once, though, I promised myself I'd do my best to make sure that it never happened again. The "primary copy" of all my data lives on the individual machines (my workstation, primarily, but there's a bit on my main laptop too) but there's also a copy of it all on a server out in the garage as well as yet another server (see above) that I have in an ISP's facility nearby. There's yet another copy of a small fraction of my files (the "really, really, really important stuff") that's sitting in AWS (via tarsnap) as well.

Some folks are satisfied with a copy of their family photos copied onto a flash drive and tossed into a drawer or an external USB drive permanently sitting on the desk next to their computer. I know of several small companies in my area that thought they were safe with an external USB drive connected to their server... until they got hit with ransomware.

My laptop has a pair of mirrored SSDs, my workstation has a pair of mirrored SSDs and a pair of mirrored "spinners". The server in the garage (my "first backup") has RAID10. That box at the ISP has mirrored SSDs plus a "raidz2" that the backups live on. Some of us just want a little bit more reassurance than others. :-)

Re: Xz format inadequate for long-term archiving (2017)

#69

Post by pmoriarty »

jlgaddis wrote: post #68
pmoriarty wrote: post #51

Earlier quoted context omitted.

"You haven't?" No I don't, because it's a waste of space and money. Using par2 and/or dvdisaster I can archive a lot more files on to the same archival media and still get enough redundancy to feel secure. "Storage is cheap." Cheap is relative. Are you really buying an extra 2 TB's of storage to archive 1 TB of data, because that's what you'd need to do to archive 2 copies of each file. That's a huge waste of space a…

> Are you really buying an extra 2 TB's of storage to archive 1 TB of data ... $ sudo zpool get size zdata NAME PROPERTY VALUE SOURCE zdata size 21.8T - Yep. It's fine that you "feel secure" with your current backup regimen -- and I certainly hope you never lose any important data. After losing data once, though, I promised myself I'd do my best to make sure that it never happened again. The "primary copy" of all my…

Good for you. If you can afford it, go for it. But I'd still use something like par2 over each of your backups.

Re: Xz format inadequate for long-term archiving (2017)

#70

Post by moviuro »

doubledad222 wrote: post #13

Thank you for sharing this. I am in charge of archiving the family files - pictures, video, art projects, email. I want it available through the aging of standards and protected against the bitrot of aging hard drives. I'll be converting any xz archives I get into a better format.

Mix and match, according to criticity and max affordable data loss: multiple locations, multiple solutions, multiple local copies (e.g. one cloud solution + DVD + NAS). See: https://www.backblaze.com/blog/the-3-2-1-backup-strategy/
Post reply on HN