Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ptar: Replacing .tgz for petabyte-scale S3 archives

plakar.io

Post reply on HN
Print view View original

31–40 of 65 posts

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#31

Post by foota »

coredog64 wrote: post #25
kjellsbells wrote: post #17

Earlier quoted context omitted.

Perhaps reframe the problem not as data loss because S3's technical infrastructure failed but because of one of the many other ways that data can get zapped or that you might need it. For example: - Employee goes rogue and nukes buckets. - Code fault quietly deletes data, or doesnt store it like you thought. - State entity demands access to data, and you'd rather give them a tape than your S3 keys. I agree that with…

A fun one I've seen before: Your encrypted content reused a KMS key that was provisioned by a temporary CloudFormation stack and got torn down months ago.

Accidental crypto shredding? Oof.

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#33

Post by msgodel »

winrid wrote: post #12

If you zoom in on your site before the cookies banner pops up you are stuck with just "Hi, we're cookies!" stuck on the screen and can't zoom out out

You don't even need a banner like this unless you have third party cookies which there are no good reasons for.

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#34

Post by kazinator »

ac29 wrote: post #32

Are people really using gzip in 2025 for new projects? Zstd has been widely available for a long time. Debian, which is pretty conservative with new software, has shipped zstd since at least stretch (released 2017).

I integrated gzip into TXR Lisp in 2022. I evaluated all the choices and went with that one because of:

- tiny code size; - widely used standard; - fast compression and decompression.

And it also beat Zstandard on compressing TXR Lisp .tlo files by a non-negligible margin. I can reproduce that today:

  $ zstd -o compiler.tlo.zstd stdlib/compiler.tlo
  stdlib/compiler.tlo  : 25.60%   (250146 =>  64037 bytes, compiler.tlo.zstd)
  $ gzip -c > compiler.tlo.gzip stdlib/compiler.tlo
  $ ls -l compiler.tlo.*
  -rw-rw-r-- 1 kaz kaz 60455 Jul  8 21:17 compiler.tlo.gzip
  -rw-rw-r-- 1 kaz kaz 64037 Jul  8 17:43 compiler.tlo.zstd
The .gzip file is 0.944 as large as the .zstd file.

So for this use case, gzip is faster (zstd has only decompression that is fast), compresses better and has way smaller code footprint.

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#35

Post by jamesfinlayson »

nemothekid wrote: post #2

> By contrast, S3 buckets are rarely backed up (a rather short-sighted approach for mission-critical cloud data), and even one-off archives are rarely done. This is a complete aside, but how often are people backing up data to something other than S3? What I mean is it some piece of data is on S3, do people have a contingency for "S3 failing". S3 is so durable in my mind now that I really only imagine having an "S3 b…

I worked at a place that uses AWS Backup - which I assume under the hood uses S3.

The backups themselves were off-limits to regular employees though - only the team that managed AWS could edit or delete the backups.

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#36

Post by xyzzy123 »

toomuchtodo wrote: post #30
kjellsbells wrote: post #17

Earlier quoted context omitted.

Perhaps reframe the problem not as data loss because S3's technical infrastructure failed but because of one of the many other ways that data can get zapped or that you might need it. For example: - Employee goes rogue and nukes buckets. - Code fault quietly deletes data, or doesnt store it like you thought. - State entity demands access to data, and you'd rather give them a tape than your S3 keys. I agree that with…

This is solved for using versioning with MFA for delete or corruption risk, S3 export if required to provide a copy. Data can also be replicated to a write only bucket in another account, with only the ability to replicate. https://docs.aws.amazon.com/AmazonS3/latest/userguide/MultiF... https://docs.aws.amazon.com/AmazonS3/latest/userguide/object...

Yep, for many applications, versioning is the lightweight solve.

But.. aws backup is still nice, if a bit heavy. I like common workflows to restore all stuff (ddbs, managed dbs, buckets etc) to a common point in time. Also, one of the under-appreciated causes of massive data loss is subtly incorrect lifecycle policies. Backup can save you here even when other techniques may not.

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#37

Post by fpoling »

kjellsbells wrote: post #17
nemothekid wrote: post #2

> By contrast, S3 buckets are rarely backed up (a rather short-sighted approach for mission-critical cloud data), and even one-off archives are rarely done. This is a complete aside, but how often are people backing up data to something other than S3? What I mean is it some piece of data is on S3, do people have a contingency for "S3 failing". S3 is so durable in my mind now that I really only imagine having an "S3 b…

Perhaps reframe the problem not as data loss because S3's technical infrastructure failed but because of one of the many other ways that data can get zapped or that you might need it. For example: - Employee goes rogue and nukes buckets. - Code fault quietly deletes data, or doesnt store it like you thought. - State entity demands access to data, and you'd rather give them a tape than your S3 keys. I agree that with…

S3 provides an object lock in compliance mode when nobody at the organization including its admins can delete objects during the specified period.

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#38

Post by FooBarWidget »

kjellsbells wrote: post #17
nemothekid wrote: post #2

> By contrast, S3 buckets are rarely backed up (a rather short-sighted approach for mission-critical cloud data), and even one-off archives are rarely done. This is a complete aside, but how often are people backing up data to something other than S3? What I mean is it some piece of data is on S3, do people have a contingency for "S3 failing". S3 is so durable in my mind now that I really only imagine having an "S3 b…

Perhaps reframe the problem not as data loss because S3's technical infrastructure failed but because of one of the many other ways that data can get zapped or that you might need it. For example: - Employee goes rogue and nukes buckets. - Code fault quietly deletes data, or doesnt store it like you thought. - State entity demands access to data, and you'd rather give them a tape than your S3 keys. I agree that with…

Or: AWS closes your account with a vague reason ("you violated our terms, we won't tell you which one") with no way to appeal.

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#39

Post by fpoling »

icedchai wrote: post #20
charcircuit wrote: post #19

Earlier quoted context omitted.

It doesn't let you.

It can be done if you delete the versions. You’ll need to use the aws cli.

It cannot be done if S3 objects use the object lock in compliance mode. Such objects cannot be altered in any way and the bucket cannot be deleted until the lock expires .

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#40

Post by Scaevolus »

Having the entire backup as a single file is interesting, but does it matter?

Restic has a similar featureset (deduplicated encrypted backups), but almost certainly has better incremental performance for complex use cases like storing X daily backups, Y weekly backups, etc. At the same time, it struggles with RAM usage when handling even 1TB of data, and presumably ptar has better scaling at that size.

Post reply on HN