Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ptar: Replacing .tgz for petabyte-scale S3 archives

plakar.io

Post reply on HN
Print view View original

11–20 of 65 posts

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#11

Post by treve »

nemothekid wrote: post #2

> By contrast, S3 buckets are rarely backed up (a rather short-sighted approach for mission-critical cloud data), and even one-off archives are rarely done. This is a complete aside, but how often are people backing up data to something other than S3? What I mean is it some piece of data is on S3, do people have a contingency for "S3 failing". S3 is so durable in my mind now that I really only imagine having an "S3 b…

We can get everything back except data. It feels silly to take the risk _not_ to if you're somewhat established.

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#13

Post by firesteelrain »

nemothekid wrote: post #2

> By contrast, S3 buckets are rarely backed up (a rather short-sighted approach for mission-critical cloud data), and even one-off archives are rarely done. This is a complete aside, but how often are people backing up data to something other than S3? What I mean is it some piece of data is on S3, do people have a contingency for "S3 failing". S3 is so durable in my mind now that I really only imagine having an "S3 b…

My HOA uses a SmartNAS in addition to S3. And we aren’t a huge operation.

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#14

Post by imglorp »

nemothekid wrote: post #10
SteveNuts wrote: post #3

Earlier quoted context omitted.

Yes, I am paranoid of S3. Not only could a once in a lifetime event happen, an attacker could get in and delete all my data. Data could be accidentally deleted. Corrupted data could be written...

> Data could be accidentally deleted. Corrupted data could be written... You guys should really have versioning enabled. Now if someone deleted your data and all the versions, that could be possible, but that would take real effort and would like be malicious.

Nobody mentioned the case where you get locked out of your cloud provider with no humans to speak to, or your account gets deleted by the algorithm. Both happen routinely and we only hear about it when the victim takes to the socials.

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#15

Post by zzo38computer »

nemothekid wrote: post #2

> By contrast, S3 buckets are rarely backed up (a rather short-sighted approach for mission-critical cloud data), and even one-off archives are rarely done. This is a complete aside, but how often are people backing up data to something other than S3? What I mean is it some piece of data is on S3, do people have a contingency for "S3 failing". S3 is so durable in my mind now that I really only imagine having an "S3 b…

I prefer to store backups on "write once read many" media, such as DVDs. However, having multiple backups would be helpful.

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#16

Post by tuckerman »

nemothekid wrote: post #8
joshka wrote: post #5

Earlier quoted context omitted.

Backups don't just protect you from durability issues. They protect you from accidental deletion, malware, and even just snapshots of what something looked at a particular time etc. The context that this article suggests is that if your S3 bucket is your primary storage, then it's possible that you're not thinking about where the second copy of your data should belong.

> They protect you from accidental deletion, malware, and even just snapshots of what something looked at a particular time etc. S3 with versioning enabled provides this. I'm not being naive when I say S3 really provides everything you might need. Its my observation over the last 13 years, dealing with tons of fires, that there has every been a situation where I couldn't retrieve something from S3. Legally you might…

Insider risk is a potential reason. If someone acquires root in your AWS account, having a backup might give you options to dealing with blackmail or even malicious deletion after it happens.

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#17

Post by kjellsbells »

nemothekid wrote: post #2

> By contrast, S3 buckets are rarely backed up (a rather short-sighted approach for mission-critical cloud data), and even one-off archives are rarely done. This is a complete aside, but how often are people backing up data to something other than S3? What I mean is it some piece of data is on S3, do people have a contingency for "S3 failing". S3 is so durable in my mind now that I really only imagine having an "S3 b…

Perhaps reframe the problem not as data loss because S3's technical infrastructure failed but because of one of the many other ways that data can get zapped or that you might need it. For example:

- Employee goes rogue and nukes buckets.

- Code fault quietly deletes data, or doesnt store it like you thought.

- State entity demands access to data, and you'd rather give them a tape than your S3 keys.

I agree that with eleven-nines or whatever it is of availability, a write to S3 is not going to disappoint you, but most data losses are more about policy and personnel than infrastructure failures.

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#18

Post by hxtk »

nemothekid wrote: post #2

> By contrast, S3 buckets are rarely backed up (a rather short-sighted approach for mission-critical cloud data), and even one-off archives are rarely done. This is a complete aside, but how often are people backing up data to something other than S3? What I mean is it some piece of data is on S3, do people have a contingency for "S3 failing". S3 is so durable in my mind now that I really only imagine having an "S3 b…

I’ve worked on a project with strict legal record-keeping requirements that had a plan for the primary AWS region literally getting nuked. But that was the only contingency in our book of plans that really required the S3 backup. We generally assumed that as long as the region still existed, S3 still had everything we put in it.

Of course, since we had the backups, restoration of individual objects would’ve been possible, but we would’ve needed to do it by hand.

Re: Ptar: Replacing .tgz for petabyte-scale S3 archives

#19

Post by charcircuit »

icedchai wrote: post #9
nemothekid wrote: post #8

Earlier quoted context omitted.

> They protect you from accidental deletion, malware, and even just snapshots of what something looked at a particular time etc. S3 with versioning enabled provides this. I'm not being naive when I say S3 really provides everything you might need. Its my observation over the last 13 years, dealing with tons of fires, that there has every been a situation where I couldn't retrieve something from S3. Legally you might…

What if someone deletes a bucket? Then all your versioning is gone...

It doesn't let you.
Post reply on HN