Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tarsnap outage postmortem

mail.tarsnap.com

Post reply on HN
Print view View original

291–300 of 338 posts

Re: Tarsnap outage postmortem

#291

Post by lanstin »

Roark66 wrote: post #190
VWWHFSfQ wrote: post #120

Earlier quoted context omitted.

Honestly, whose data isn't "super important"?. All my data is super important. Even the crap I just throw on my Google drive. I want to keep it. What is this mythical unimportant data that people still want to back up?

>What is this mythical unimportant data that people still want to back up? I have lots of stuff like this. Often it is easier to just back up an entire folder than go through sub/sub folders separating stuff into: important, not very important. Storage costs are low enough to just backup everything (almost). Also, one often doesn't know what may be important/useful in future. For example a couple of years ago I had t…

I have stuff like that on a hard drive in my home, on a persistent storage volume from Linode, and on Dropbox.

Re: Tarsnap outage postmortem

#292

Post by rssoconnor »

ufmace wrote: post #290
rssoconnor wrote: post #140

Earlier quoted context omitted.

I use tarsnap because the asymmetric crypto means I can give my cron job authorization to create backups, but it doesn't have authorization to read or delete(!) backups. This ability is critical to prevent a compromised system from having its data wiped and having all backups wiped as well. I haven't been able to figure out how to do this in any other system. But if someone has a tutorial, I am all ears.

IMO AWS S3 works fine for this: * Create a S3 bucket and enable versioning * Create a new user and give it only s3:PutObject on your new bucket * Create an auth keypair for that user and put it on your server Now any server compromise that gets those keys can only add new data to your backup bucket, and can't read, overwrite, or delete any previous backup. There's no dedup, so that could be a deal-breaker. There's al…

Deduplication isn't necessarily a deal breaker. Let's see.

My main machine is currently storing 1.6 TB (compressed) of total archives with tarsnap, but only 33 GB (compressed) of unique data within those archives. So if S3 is 50x cheaper, then not having deduplication would be a wash.

However other comments here suggest that S3 is only 10x cheaper.

Re: Tarsnap outage postmortem

#293

Post by saurik »

molsongolden wrote: post #275
hunter2_ wrote: post #267

Earlier quoted context omitted.

Are you suggesting that those who build enterprises don't have time for kids? Seems plausible, but is the difference in lifestyle so consistently prevalent as to be stereotypical? Elon has 10!

Raising the toddler to have some help running the business.

Not just help: there is now a clear heir to take over if (the gods forbid) cperciva ever succumbs to illness or is defeated in battle.

Re: Tarsnap outage postmortem

#294

Post by hgsgm »

idlewords wrote: post #220
jacquesm wrote: post #192

Earlier quoted context omitted.

In future postmortems (of which I hope there will be very few or even none) you may want to spell out your 'lessons learned' to show why particular items will never recur.

It always amuses me how people want reassurance that the next crisis will be a fresh, new problem, and not one the person can demonstrably solve. A lot of 'lessons learned' analysis boils down to this: in order to prevent a recurrence of X, we introduced complex subsystem Y, the unexpected effects of which you can read about in our next post-mortem.

You'd love my team's recent postmortem, featuring the comment "action items haeve been copied from the previous postmortem".

Re: Tarsnap outage postmortem

#295

Post by Phlarp »

lifeisstillgood wrote: post #207
justin_oaks wrote: post #206

Earlier quoted context omitted.

I know it's a joke, but I think if an SLA involved putting a CTO in stocks and throwing eggs at him then that'd encourage me to sign up for the service. Especially if the video of it were posted after every incident. Instead we get refunded some pitiful amount when our business is seriously disrupted for an extended period of time.

:-) My youngest once found some sort of chocolate drops called "unicorn poo" - which seems a more ironic thing to chuck at CTOs !

Don't let the CTO be a scapegoat. Entire executive leadership, all board members and the 5 largest shareholders.

Re: Tarsnap outage postmortem

#296

Post by devonkim »

twic wrote: post #252
stigz wrote: post #9

Earlier quoted context omitted.

Why would I use your service over restic? God bless you Colin, but reading this, it appears you're the only one in charge of the infrastructure for this service. I'm glad you're clear about no SLA, but this seems like a big liability between me and my backups.

> God bless you Colin, but reading this, it appears you're the only one in charge of the infrastructure for this service Hence the toddler.

I am really confused by the communication thread and am interpreting that the toddler is somehow in charge of running the infrastructure as a joke. Yet I can’t see it as either a joke or serious.

I’m a native English speaker but sometimes I swear I’m losing grasp on communication in the Internet age and am sincerely trying to understand this all.

Re: Tarsnap outage postmortem

#297

Post by NhanH »

devonkim wrote: post #296
twic wrote: post #252

Earlier quoted context omitted.

> God bless you Colin, but reading this, it appears you're the only one in charge of the infrastructure for this service Hence the toddler.

I am really confused by the communication thread and am interpreting that the toddler is somehow in charge of running the infrastructure as a joke. Yet I can’t see it as either a joke or serious. I’m a native English speaker but sometimes I swear I’m losing grasp on communication in the Internet age and am sincerely trying to understand this all.

The joke is that the toddler is for future maintenance, not now.

Re: Tarsnap outage postmortem

#298

Post by nextaccountic »

cperciva wrote: post #3

blinks Ok, I really wasn't expecting this to land at the top of HN. I'd love to stick around to answer any questions people have, but it's 10PM and my toddler decided to go to bed at 5PM... so if I'm lucky I can get about 4 hours of sleep before she decides that it's time to get up. I'll check in and answer questions in the morning.

> the central Tarsnap server (hosted in Amazon's EC2 us-east-1 region)

What prevents you to distribute load among other regions?

(Also: did you ever think about abandoning AWS?)

Re: Tarsnap outage postmortem

#299

Post by rlt »

cperciva wrote: post #3

blinks Ok, I really wasn't expecting this to land at the top of HN. I'd love to stick around to answer any questions people have, but it's 10PM and my toddler decided to go to bed at 5PM... so if I'm lucky I can get about 4 hours of sleep before she decides that it's time to get up. I'll check in and answer questions in the morning.

Nice write up. A couple questions:

- The use of “I” begs the question: what’s the “bus factor” of Tarsnap? If you were unavailable, temporarily or permanently, what are the contingency plans?

- Will you be making any other changes to improve the recovery time, or did the system mostly function as designed? For example having a hot spare central server?

Re: Tarsnap outage postmortem

#300

Post by cperciva »

molsongolden wrote: post #275
hunter2_ wrote: post #267

Earlier quoted context omitted.

Are you suggesting that those who build enterprises don't have time for kids? Seems plausible, but is the difference in lifestyle so consistently prevalent as to be stereotypical? Elon has 10!

Raising the toddler to have some help running the business.

Might take a while. Tarsnap has never had an employee without a doctorate. She's a very bright girl but I'll be surprised if she gets her doctorate before 2040.
Post reply on HN