Ptar: Replacing .tgz for petabyte-scale S3 archives
1–10 of 65 posts
Re: Ptar: Replacing .tgz for petabyte-scale S3 archives
#2>By contrast, S3 buckets are rarely backed up (a rather short-sighted approach for mission-critical cloud data), and even one-off archives are rarely done.
This is a complete aside, but how often are people backing up data to something other than S3? What I mean is it some piece of data is on S3, do people have a contingency for "S3 failing".
S3 is so durable in my mind now that I really only imagine having an "S3 backup" if (1) I had an existing system (e.g. tapes), or (2) I need multi-cloud redundancy. Other than that, once I assume something is in S3, I confident it's safe.
Obviously this was built over years (decades?) or reliability, and if your DRP requires alternatives, you should do them, but is anyone realistically paranoid about S3?
Re: Ptar: Replacing .tgz for petabyte-scale S3 archives
#3> By contrast, S3 buckets are rarely backed up (a rather short-sighted approach for mission-critical cloud data), and even one-off archives are rarely done. This is a complete aside, but how often are people backing up data to something other than S3? What I mean is it some piece of data is on S3, do people have a contingency for "S3 failing". S3 is so durable in my mind now that I really only imagine having an "S3 b…
Yes, I am paranoid of S3. Not only could a once in a lifetime event happen, an attacker could get in and delete all my data. Data could be accidentally deleted. Corrupted data could be written...
Re: Ptar: Replacing .tgz for petabyte-scale S3 archives
#4> By contrast, S3 buckets are rarely backed up (a rather short-sighted approach for mission-critical cloud data), and even one-off archives are rarely done. This is a complete aside, but how often are people backing up data to something other than S3? What I mean is it some piece of data is on S3, do people have a contingency for "S3 failing". S3 is so durable in my mind now that I really only imagine having an "S3 b…
But don't make the same mistake people make with RAID. "More durable" doesn't mean "backup".
What if somebody deletes the file? What if it got corrupted for a problem in one of your processes? What if your API key falls in the wrong hands?
Re: Ptar: Replacing .tgz for petabyte-scale S3 archives
#5> By contrast, S3 buckets are rarely backed up (a rather short-sighted approach for mission-critical cloud data), and even one-off archives are rarely done. This is a complete aside, but how often are people backing up data to something other than S3? What I mean is it some piece of data is on S3, do people have a contingency for "S3 failing". S3 is so durable in my mind now that I really only imagine having an "S3 b…
Backups don't just protect you from durability issues. They protect you from accidental deletion, malware, and even just snapshots of what something looked at a particular time etc.
The context that this article suggests is that if your S3 bucket is your primary storage, then it's possible that you're not thinking about where the second copy of your data should belong.
Re: Ptar: Replacing .tgz for petabyte-scale S3 archives
#6> By contrast, S3 buckets are rarely backed up (a rather short-sighted approach for mission-critical cloud data), and even one-off archives are rarely done. This is a complete aside, but how often are people backing up data to something other than S3? What I mean is it some piece of data is on S3, do people have a contingency for "S3 failing". S3 is so durable in my mind now that I really only imagine having an "S3 b…
Yes, I am paranoid of S3. Not only could a once in a lifetime event happen, an attacker could get in and delete all my data. Data could be accidentally deleted. Corrupted data could be written...
Then 3 steps.
1. Use tarsnap so there's an encryption and a management layer.
2. Use a second service so there's redundancy and no SPoF.
3. Keep cryptographic signatures (not hashes) of each backup job in something like a WORM blockchain KVS.
Re: Ptar: Replacing .tgz for petabyte-scale S3 archives
#7> By contrast, S3 buckets are rarely backed up (a rather short-sighted approach for mission-critical cloud data), and even one-off archives are rarely done. This is a complete aside, but how often are people backing up data to something other than S3? What I mean is it some piece of data is on S3, do people have a contingency for "S3 failing". S3 is so durable in my mind now that I really only imagine having an "S3 b…
But don't make the same mistake people make with RAID. "More durable" doesn't mean "backup". What if somebody deletes the file? What if it got corrupted for a problem in one of your processes? What if your API key falls in the wrong hands?
Yes - backups also protect against someone doing a `rm -rf /*` by accident. However, I don't think I've created an S3 bucket without versioning enabled for years. If someone deletes the file, or the file gets corrupted - I just restore a pervious version.
I don't want to suggest that people should place all their eggs in one basket - it's obviously irresponsible. However, S3 (and versioning) has been the "final storage" for years now. I can only imagine a catastrophic situation like an entire s3 region blowing up. And I'm sure a disgruntled employee could do a lot of damage as well.
Re: Ptar: Replacing .tgz for petabyte-scale S3 archives
#8> By contrast, S3 buckets are rarely backed up (a rather short-sighted approach for mission-critical cloud data), and even one-off archives are rarely done. This is a complete aside, but how often are people backing up data to something other than S3? What I mean is it some piece of data is on S3, do people have a contingency for "S3 failing". S3 is so durable in my mind now that I really only imagine having an "S3 b…
Backups don't just protect you from durability issues. They protect you from accidental deletion, malware, and even just snapshots of what something looked at a particular time etc. The context that this article suggests is that if your S3 bucket is your primary storage, then it's possible that you're not thinking about where the second copy of your data should belong.
>They protect you from accidental deletion, malware, and even just snapshots of what something looked at a particular time etc.
S3 with versioning enabled provides this. I'm not being naive when I say S3 really provides everything you might need. Its my observation over the last 13 years, dealing with tons of fires, that there has every been a situation where I couldn't retrieve something from S3.
Legally you might need an alternative. Going multi-cloud doesn't hurt - after all I do it. But practically? I don't think I would lose sleep if someone told me they only back up to S3.
Re: Ptar: Replacing .tgz for petabyte-scale S3 archives
#9Earlier quoted context omitted.
Backups don't just protect you from durability issues. They protect you from accidental deletion, malware, and even just snapshots of what something looked at a particular time etc. The context that this article suggests is that if your S3 bucket is your primary storage, then it's possible that you're not thinking about where the second copy of your data should belong.
> They protect you from accidental deletion, malware, and even just snapshots of what something looked at a particular time etc. S3 with versioning enabled provides this. I'm not being naive when I say S3 really provides everything you might need. Its my observation over the last 13 years, dealing with tons of fires, that there has every been a situation where I couldn't retrieve something from S3. Legally you might…
What if someone deletes a bucket? Then all your versioning is gone...
Re: Ptar: Replacing .tgz for petabyte-scale S3 archives
#10> By contrast, S3 buckets are rarely backed up (a rather short-sighted approach for mission-critical cloud data), and even one-off archives are rarely done. This is a complete aside, but how often are people backing up data to something other than S3? What I mean is it some piece of data is on S3, do people have a contingency for "S3 failing". S3 is so durable in my mind now that I really only imagine having an "S3 b…
Yes, I am paranoid of S3. Not only could a once in a lifetime event happen, an attacker could get in and delete all my data. Data could be accidentally deleted. Corrupted data could be written...
>Data could be accidentally deleted. Corrupted data could be written...
You guys should really have versioning enabled. Now if someone deleted your data and all the versions, that could be possible, but that would take real effort and would like be malicious.