Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tarsnap outage postmortem

mail.tarsnap.com

Post reply on HN
Print view View original

261–270 of 338 posts

Re: Tarsnap outage postmortem

#261

Post by gus_massa »

cperciva wrote: post #247
idlewords wrote: post #213

Hats off to you for an honest postmortem and your capable handling of a difficult situation. The only remark I would offer is with respect to sleep deprivation—when you're the only person who can fix a problem, there's no shame in trading some additional outage time for a fresh mind. Though it feels weird to go nap when all the klaxons are blaring, problems are too easy to compound under the combination of adrenaline…

Don't worry, I had a couple naps in there. "This seems to be running smoothly but it will take several more hours; I'll set my alarm to wake me up in two hours and have a nap" is part of why I didn't notice the second step was unnecessarily I/O bound.

IIUC the process had a few steps were you only had to wait while data was transferred or processed for long times. They were probably useful to take a nap or eat or just drink more coffee.

Re: Tarsnap outage postmortem

#262

Post by dryrun »

Alupis wrote: post #253
andrewmunsell wrote: post #242

Earlier quoted context omitted.

I hate to bring this up, but what about the bus factor? If Colin is physically unable to continue maintaining the service and something like this happens again, how will anyone be able to get their data out? It's not really a concern about the service Tarsnap provides today

There's an old Sys Admin saying (perhaps from Allan Jude of ScaleEngine) that goes something like "if your data doesn't exist in at least three places, it doesn't actually exist at all..." That is to say, if Tarsnap is the only place you've keeping sensitive/important data, then you're "not doing it right" as a backup. Things happen... your hard drive can die suddenly, and a data center bursts into flames all on the…

I feel like ovh will never stop earing about this. This has been, frankly, a traumatic event for many sysadmins I believe, and one that was shared by many from the same source, which is quite different from the standard variation of "that time when I erased the production database" (looking at you gitlab, but also at myself!). I mean, at this point it's between a legend and a warning tale and I don't know what else to call it. A bad Wednesday probably.

Re: Tarsnap outage postmortem

#263

Post by catiopatio »

zetalyrae wrote: post #112
amluto wrote: post #104

Earlier quoted context omitted.

> This would benefit from being further from the metal. How, exactly, is that a good thing?

How is not rolling your own database a good thing? Mainly because the business of tarsnap is 1) encrypted 2) backups, not building a database storage engine.

Implementing client-side encrypted, deduplicated, snapshot-enabled backups with server-mediated access control inherently requires building a minimal storage engine to represent your opaque log-structured data.

Embedding the log-structured representation of user data in Postgres would increase complexity and overhead without offering significant resiliency or recoverability advantages — in fact, quite the opposite.

Re: Tarsnap outage postmortem

#264

Post by cperciva »

rsync wrote: post #248
deathanatos wrote: post #238

> Following my ill-defined "Tarsnap doesn't have an SLA but I'll give people credits for outages when it seems fair" policy, on 2023-07-13 (after some dust settled and I caught up on some sleep) I credited everyone's Tarsnap accounts with 50% of a month's storage costs. This speaks volumes to me about what kind of person Percival is; that credit would appear to be generously on the "make customer whole" side of the f…

"Thanks for being a beacon of light in the dark." That's well put. It makes me very happy to live in a world where tarsnap exists and is priced in picodollars.

For the record, I'm happy to live in a world where rsync.net exists. I've pointed quite a few customers in your direction over the years, when tarsnap hasn't been suitable for their needs for a variety of reasons.

Re: Tarsnap outage postmortem

#265

Post by jacquesm »

cperciva wrote: post #232
jacquesm wrote: post #192

Earlier quoted context omitted.

In future postmortems (of which I hope there will be very few or even none) you may want to spell out your 'lessons learned' to show why particular items will never recur.

Yeah, I was going to do that but it was getting late, I wanted to get some sleep, and the post-mortem had already been waiting far too long to be sent out. The main lesson learned was "rehearse this process at least once a year".

Agreed, that's the big one. But also: when sleep deprived: take a nap!

Re: Tarsnap outage postmortem

#266

Post by jacquesm »

idlewords wrote: post #220
jacquesm wrote: post #192

Earlier quoted context omitted.

In future postmortems (of which I hope there will be very few or even none) you may want to spell out your 'lessons learned' to show why particular items will never recur.

It always amuses me how people want reassurance that the next crisis will be a fresh, new problem, and not one the person can demonstrably solve. A lot of 'lessons learned' analysis boils down to this: in order to prevent a recurrence of X, we introduced complex subsystem Y, the unexpected effects of which you can read about in our next post-mortem.

That was entirely not what I was getting at and is a cheap shot that is well beneath you, especially because I suspect that you know that that wasn't what I was getting at.

Re: Tarsnap outage postmortem

#267

Post by hunter2_ »

twic wrote: post #252
stigz wrote: post #9

Earlier quoted context omitted.

Why would I use your service over restic? God bless you Colin, but reading this, it appears you're the only one in charge of the infrastructure for this service. I'm glad you're clear about no SLA, but this seems like a big liability between me and my backups.

> God bless you Colin, but reading this, it appears you're the only one in charge of the infrastructure for this service Hence the toddler.

Are you suggesting that those who build enterprises don't have time for kids? Seems plausible, but is the difference in lifestyle so consistently prevalent as to be stereotypical? Elon has 10!

Re: Tarsnap outage postmortem

#268

Post by e63f67dd-065b »

cperciva wrote: post #3

blinks Ok, I really wasn't expecting this to land at the top of HN. I'd love to stick around to answer any questions people have, but it's 10PM and my toddler decided to go to bed at 5PM... so if I'm lucky I can get about 4 hours of sleep before she decides that it's time to get up. I'll check in and answer questions in the morning.

Can you say a bit more about the log-structured S3 filesystem? I wrote something very similar recently (https://github.com/isaackhor/objectfs) and I'm curious what made you settle on that architecture. The closest thing I know of that's similar is Nvidia's ProxyFS (https://github.com/NVIDIA/proxyfs)

Re: Tarsnap outage postmortem

#269

Post by nemothekid »

tptacek wrote: post #195
mekster wrote: post #29

Earlier quoted context omitted.

It's insane. Not sure how anyone can accept such a rip off pricing. Tarsnap : $0.25 / GB storage, $0.25 / GB bandwidth cost rsync.net : $0.015 / GB storage, no bandwidth cost s3 : $0.023 / GB storage, some complicated bandwidth pricing If tarsnap is built on top of s3, they're charging 10 times for the storage cost. Easy money from the uninformed?

Yes. That's the thing about Tarsnap, a service with a TikZ diagram on its front page, built around a Unix utility, that meters in picodollars. It's meant to bilk money from uninformed mom-and-pop backup users.

I'm having a hard time to believe that anyone remotely interested in Tarsnap's value prop is also an "uninformed mom-and-pop".

This "uninformed mom-and-pop" is potentially compiling the client application from source, but can't do basic math to compare tarsnap's pricing to the top 20 or so competitors that rank above tarsnap in SEO?

Re: Tarsnap outage postmortem

#270

Post by tptacek »

nemothekid wrote: post #269
tptacek wrote: post #195

Earlier quoted context omitted.

Yes. That's the thing about Tarsnap, a service with a TikZ diagram on its front page, built around a Unix utility, that meters in picodollars. It's meant to bilk money from uninformed mom-and-pop backup users.

I'm having a hard time to believe that anyone remotely interested in Tarsnap's value prop is also an "uninformed mom-and-pop". This "uninformed mom-and-pop" is potentially compiling the client application from source, but can't do basic math to compare tarsnap's pricing to the top 20 or so competitors that rank above tarsnap in SEO?

That was sarcasm. :)
Post reply on HN