Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ptar: Replacing .tgz for petabyte-scale S3 archives

plakar.io

Post reply on HN
Print view View original

51–60 of 65 posts

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#51

Post by mrflop »

ahofmann wrote: post #49

I'm trying to evaluate what plakar is. Is it like restic, Borgbackup, Kopia?

Yes Plakar works much like Restic and Kopia: it takes content-addressed, encrypted and deduplicated snapshots and offers efficient incremental backups via a simple CLI. Under the hood, its Kloset engine splits data into encrypted, compressed chunks. Plakar main strengths:

UI: In addition to a simple Unix-style CLI, Plakar provides an web interface and API for monitoring, browsing snapshots

Data-agnostic snapshots: Plakar’s Kloset engine captures any structured data—filesystems, databases, applications—not just files, by organizing them into self-describing snapshots

Source/target decoupling: You can back up from one system (e.g. a local filesystem) and restore to another (e.g. an S3 bucket) using pluggable source and target connectors

Universal storage backends: Storage connectors let you persist encrypted, compressed chunks to local filesystems, SFTP servers or S3-compatible object stores (and more)—all via a unified interface

Extreme scale with low RAM: A virtual filesystem with lazy loading and backpressure-aware parallelism keeps memory use minimal, even on very large datasets

Network- and egress-optimized: Advanced client-side deduplication and compression dramatically cut storage and network transfer costs—ideal for inter-cloud or cross-provider migrations

Online maintenance: you don't need to stop you backup to free some space

ptar...

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#52

Post by fpoling »

icedchai wrote: post #48
fpoling wrote: post #39

Earlier quoted context omitted.

It cannot be done if S3 objects use the object lock in compliance mode. Such objects cannot be altered in any way and the bucket cannot be deleted until the lock expires .

Good to know! I’ve never used that feature.

Note that with such lock mistakes can be costly. If you put into S3 several terabytes by mistake and set the compliance lock duration for 2 years, you will have to pay for that storage for 2 years.

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#53

Post by chungy »

mrflop wrote: post #50
chungy wrote: post #21

Another similar archive format is WIM, the thing created by Microsoft for the Windows Vista (and newer) installer; an open source implementation is at: https://wimlib.net/ It offers similar deduplication, indexing, per-file compression, and versioning advantages

But it works only for Windows, right?

No, it works on many OSes. That's the point of linking to wimlib :)

It even supports Unix metadata!

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#54

Post by mrflop »

xyzzy123 wrote: post #36
toomuchtodo wrote: post #30

Earlier quoted context omitted.

This is solved for using versioning with MFA for delete or corruption risk, S3 export if required to provide a copy. Data can also be replicated to a write only bucket in another account, with only the ability to replicate. https://docs.aws.amazon.com/AmazonS3/latest/userguide/MultiF... https://docs.aws.amazon.com/AmazonS3/latest/userguide/object...

Yep, for many applications, versioning is the lightweight solve. But.. aws backup is still nice, if a bit heavy. I like common workflows to restore all stuff (ddbs, managed dbs, buckets etc) to a common point in time. Also, one of the under-appreciated causes of massive data loss is subtly incorrect lifecycle policies. Backup can save you here even when other techniques may not.

AWS Backup can get really pricey since you pay GB-month for every single restore point. Plakar only charges once for the initial backup and then for the small deltas on whatever cheap storage you pick.

Also, AWS Backup locks your snapshots into AWS vaults, whereas Plakar lets you push and pull backups to any backend—local disk, S3, another cloud, on-prem, etc.

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#55

Post by mrflop »

fpoling wrote: post #37
kjellsbells wrote: post #17

Earlier quoted context omitted.

Perhaps reframe the problem not as data loss because S3's technical infrastructure failed but because of one of the many other ways that data can get zapped or that you might need it. For example: - Employee goes rogue and nukes buckets. - Code fault quietly deletes data, or doesnt store it like you thought. - State entity demands access to data, and you'd rather give them a tape than your S3 keys. I agree that with…

S3 provides an object lock in compliance mode when nobody at the organization including its admins can delete objects during the specified period.

S3 buckets can just vanish for lots of reasons. With AWS’s shared-responsibility model, you’re the one who has to back up and protect your data not AWS.

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#56

Post by mrflop »

SteveNuts wrote: post #3
nemothekid wrote: post #2

> By contrast, S3 buckets are rarely backed up (a rather short-sighted approach for mission-critical cloud data), and even one-off archives are rarely done. This is a complete aside, but how often are people backing up data to something other than S3? What I mean is it some piece of data is on S3, do people have a contingency for "S3 failing". S3 is so durable in my mind now that I really only imagine having an "S3 b…

Yes, I am paranoid of S3. Not only could a once in a lifetime event happen, an attacker could get in and delete all my data. Data could be accidentally deleted. Corrupted data could be written...

That’s basically one of the reasons that led us to build Plakar.

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#58

Post by fpoling »

mrflop wrote: post #55
fpoling wrote: post #37

Earlier quoted context omitted.

S3 provides an object lock in compliance mode when nobody at the organization including its admins can delete objects during the specified period.

S3 buckets can just vanish for lots of reasons. With AWS’s shared-responsibility model, you’re the one who has to back up and protect your data not AWS.

The hold in compliance mode with AWS is accepted way to persist data that a company obliged to hold legally by US requirements.

And if your company has a sale contract with AWS the buckets cannot just vanish or AWS cannot close the account at arbitrary moment.

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#59

Post by ahofmann »

mrflop wrote: post #51
ahofmann wrote: post #49

I'm trying to evaluate what plakar is. Is it like restic, Borgbackup, Kopia?

Yes Plakar works much like Restic and Kopia: it takes content-addressed, encrypted and deduplicated snapshots and offers efficient incremental backups via a simple CLI. Under the hood, its Kloset engine splits data into encrypted, compressed chunks. Plakar main strengths: UI: In addition to a simple Unix-style CLI, Plakar provides an web interface and API for monitoring, browsing snapshots Data-agnostic snapshots: Pl…

Thank you!

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#60

Post by icedchai »

fpoling wrote: post #52
icedchai wrote: post #48

Earlier quoted context omitted.

Good to know! I’ve never used that feature.

Note that with such lock mistakes can be costly. If you put into S3 several terabytes by mistake and set the compliance lock duration for 2 years, you will have to pay for that storage for 2 years.

So not even Amazon can fix this? What if my company goes bankrupt with several TB locked up?
Post reply on HN