Live data from Hacker News

Ptar: Replacing .tgz for petabyte-scale S3 archives

plakar.io

61–65 of 65 posts

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#61
post #52

Earlier quoted context omitted.

Note that with such lock mistakes can be costly. If you put into S3 several terabytes by mistake and set the compliance lock duration for 2 years, you will have to pay for that storage for 2 years.

So not even Amazon can fix this? What if my company goes bankrupt with several TB locked up?

If you close the account with Amazon, then yes, the data can be deleted. But typically based on contract this will require notifying Amazon and will be extremely visible and can be reverted.

If the company does not pay, then the company breaches its contract and Amazon can delete the data. But typically there would be a warning period.

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#62
post #45

Earlier quoted context omitted.

I integrated gzip into TXR Lisp in 2022. I evaluated all the choices and went with that one because of: - tiny code size; - widely used standard; - fast compression and decompression. And it also beat Zstandard on compressing TXR Lisp .tlo files by a non-negligible margin. I can reproduce that today: $ zstd -o compiler.tlo.zstd stdlib/compiler.tlo stdlib/compiler.tlo : 25.60% (250146 => 64037 bytes, compiler.tlo.zstd…

I believe the default compression setting for the zstd command is biased towards speed -- maybe try -9, -13 or even -22 (max, which should probably be fine for such a small file). Not that it matters when the file is so small in the first place... I'm just saying you should be sure what you're 'benchmarking'

[deleted]

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#63
post #54

Earlier quoted context omitted.

Yep, for many applications, versioning is the lightweight solve. But.. aws backup is still nice, if a bit heavy. I like common workflows to restore all stuff (ddbs, managed dbs, buckets etc) to a common point in time. Also, one of the under-appreciated causes of massive data loss is subtly incorrect lifecycle policies. Backup can save you here even when other techniques may not.

AWS Backup can get really pricey since you pay GB-month for every single restore point. Plakar only charges once for the initial backup and then for the small deltas on whatever cheap storage you pick. Also, AWS Backup locks your snapshots into AWS vaults, whereas Plakar lets you push and pull backups to any backend—local disk, S3, another cloud, on-prem, etc.

AWS backup is a bit more nuanced than that; ideally the thing you want is N days of PITR (point in time recovery) and you want that across all your data stores (RDS dbs, buckets, dynamodb tables, etc etc), and you want to be able to restore them all to a common point in time. 7 or 30 days or PITR are common choices. It is ideal if you can perform a data restore in 1 operation since your hair may be on fire when you need to use it. In practice almost all your recovery will be from this.

The storage needed for this depends on the data change rate in your application, more or less it works like a WAL in a DB. What is annoying is that you can't really control it (for obvious reasons), and less forgivably, AWS backup is super opaque about how much is actually being used by what.

Retention of dailies / weeklies / monthlies is a different (usually compliance) concern (NOT operational, not really, if you have to restore from a monthly your business is probably already done for) and in an enterprise context you are generally prevented from using deltas for these due to enterprise policy or regulation (yeah I know it sounds crazy, reqs are getting really specific these days).

People on AWS don't generally care that they're locked in to AWS services (else.. they wouldn't be on AWS), and while cost is often a factor it is usually not the primary concern (else.. they would not be on AWS). What often IS a primary concern is knowing that their backup solution is covered under the enterprise tier AWS support they are already paying an absolute buttload for.

Also stuff like Vault lock "compliance mode" & "automated restore testing" are helpful in box-ticking scenarios.

Plakar looks awesome but I'm not sure AWS Backup customers are the right market to go for.

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#64

Earlier quoted context omitted.

Insider risk is a potential reason. If someone acquires root in your AWS account, having a backup might give you options to dealing with blackmail or even malicious deletion after it happens.

If someone acquires root in the AWS account, they likely then have access to the backups, too. Unless we're also assuming whatever is doing the backup runs in an alternate cloud and our attacker or insider somehow has access to only 1 of 2 clouds. Possible, perhaps, but contrived.

I think using a separate cloud with credentials stored in a safe (or the equivalent) isn’t that uncommon (worked places where we were nearly 100% AWS but had GCP for storing backups). You’d need to compromise/socially engineer a different set of people to get access to that.

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#65
post #43

Earlier quoted context omitted.

zstd uses a fairly low compression level by default. If you run with `zstd -19 -o compiler.tlo.zstd stdlib/compiler.tlo` you will probably get much better compression than gzip, even at its highest setting. That said, the tiny code footprint of gzip can be a real benefit. And you can usually count on gzip being available as a system library on whatever platform you're targeting, while that's often not the case for zs…

Additional datapoints: Tne Zopfli gzip-compatible compressor gets the file down to 54343. But zstd with level -19 beats that: -rw-rw-r-- 1 kaz kaz 54373 Jul 8 22:59 compiler.tlo.zopfli -rw-rw-r-- 1 kaz kaz 50102 Jul 8 17:43 compiler.tlo.zstd.19 I have no idea which is more CPU/memory intensive. For applications in which compression speed is not important (data is being prepared once to be decompressed many times), if…

Try lzip. It's about 10 times faster than zopfli though it's not gzip compatible. And it beats zstd -19 on compression.
Post reply on HN