Live data from Hacker News

Ptar: Replacing .tgz for petabyte-scale S3 archives

plakar.io

41–50 of 65 posts

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#41

Earlier quoted context omitted.

> They protect you from accidental deletion, malware, and even just snapshots of what something looked at a particular time etc. S3 with versioning enabled provides this. I'm not being naive when I say S3 really provides everything you might need. Its my observation over the last 13 years, dealing with tons of fires, that there has every been a situation where I couldn't retrieve something from S3. Legally you might…

And then Amazon kills your account. It doesn't matter how great their hardware and software is.

There is a contractual obligation on Amazon side. If they kill the account in violation of the contract, the court will force them to pay heavy damages.

Now, one can argue that courts would take time and money and a company may not afford such risk even if it is theoretical. In this case if data is that important it is stupid to keep them at AWS.

But then just write the data to tapes and store in a bank cell or whatever.

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#42

Having the entire backup as a single file is interesting, but does it matter? Restic has a similar featureset (deduplicated encrypted backups), but almost certainly has better incremental performance for complex use cases like storing X daily backups, Y weekly backups, etc. At the same time, it struggles with RAM usage when handling even 1TB of data, and presumably ptar has better scaling at that size.

> At the same time, it struggles with RAM usage when handling even 1TB of data, and presumably ptar has better scaling at that size.

There's also rustic, which supposedly is optimized for memory: https://rustic.cli.rs/docs/

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#43
post #32

Are people really using gzip in 2025 for new projects? Zstd has been widely available for a long time. Debian, which is pretty conservative with new software, has shipped zstd since at least stretch (released 2017).

I integrated gzip into TXR Lisp in 2022. I evaluated all the choices and went with that one because of: - tiny code size; - widely used standard; - fast compression and decompression. And it also beat Zstandard on compressing TXR Lisp .tlo files by a non-negligible margin. I can reproduce that today: $ zstd -o compiler.tlo.zstd stdlib/compiler.tlo stdlib/compiler.tlo : 25.60% (250146 => 64037 bytes, compiler.tlo.zstd…

zstd uses a fairly low compression level by default. If you run with `zstd -19 -o compiler.tlo.zstd stdlib/compiler.tlo` you will probably get much better compression than gzip, even at its highest setting.

That said, the tiny code footprint of gzip can be a real benefit. And you can usually count on gzip being available as a system library on whatever platform you're targeting, while that's often not the case for zstd (on iOS, for example).

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#44
post #43

Earlier quoted context omitted.

I integrated gzip into TXR Lisp in 2022. I evaluated all the choices and went with that one because of: - tiny code size; - widely used standard; - fast compression and decompression. And it also beat Zstandard on compressing TXR Lisp .tlo files by a non-negligible margin. I can reproduce that today: $ zstd -o compiler.tlo.zstd stdlib/compiler.tlo stdlib/compiler.tlo : 25.60% (250146 => 64037 bytes, compiler.tlo.zstd…

zstd uses a fairly low compression level by default. If you run with `zstd -19 -o compiler.tlo.zstd stdlib/compiler.tlo` you will probably get much better compression than gzip, even at its highest setting. That said, the tiny code footprint of gzip can be a real benefit. And you can usually count on gzip being available as a system library on whatever platform you're targeting, while that's often not the case for zs…

Additional datapoints:

Tne Zopfli gzip-compatible compressor gets the file down to 54343. But zstd with level -19 beats that:

  -rw-rw-r-- 1 kaz kaz 54373 Jul  8 22:59 compiler.tlo.zopfli
  -rw-rw-r-- 1 kaz kaz 50102 Jul  8 17:43 compiler.tlo.zstd.19
I have no idea which is more CPU/memory intensive.

For applications in which compression speed is not important (data is being prepared once to be decompressed many times), if you want the best compression and stick with gzip, Zopfli is the ticket.

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#45
post #32

Are people really using gzip in 2025 for new projects? Zstd has been widely available for a long time. Debian, which is pretty conservative with new software, has shipped zstd since at least stretch (released 2017).

I integrated gzip into TXR Lisp in 2022. I evaluated all the choices and went with that one because of: - tiny code size; - widely used standard; - fast compression and decompression. And it also beat Zstandard on compressing TXR Lisp .tlo files by a non-negligible margin. I can reproduce that today: $ zstd -o compiler.tlo.zstd stdlib/compiler.tlo stdlib/compiler.tlo : 25.60% (250146 => 64037 bytes, compiler.tlo.zstd…

I believe the default compression setting for the zstd command is biased towards speed -- maybe try -9, -13 or even -22 (max, which should probably be fine for such a small file).

Not that it matters when the file is so small in the first place... I'm just saying you should be sure what you're 'benchmarking'

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#46
post #29

They mention in the article that some people don't want to install the full Plakar backup software just to read and write ptar archives; so a dedicated open-source tool is offered for download as of yesterday: https://plakar.io/posts/2025-07-07/kapsul-a-tool-to-create-a...

Direct link to GitHub: https://github.com/PlakarKorp/kapsul

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#48
post #39

Earlier quoted context omitted.

It can be done if you delete the versions. You’ll need to use the aws cli.

It cannot be done if S3 objects use the object lock in compliance mode. Such objects cannot be altered in any way and the bucket cannot be deleted until the lock expires .

Good to know! I’ve never used that feature.

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#50
post #21

Another similar archive format is WIM, the thing created by Microsoft for the Windows Vista (and newer) installer; an open source implementation is at: https://wimlib.net/ It offers similar deduplication, indexing, per-file compression, and versioning advantages

But it works only for Windows, right?
Post reply on HN