Tarsnap outage postmortem • Hacker News Board

Re: Tarsnap outage postmortem

#241

Post by cperciva » Thu, Jul 27, 2023, 4:48 PM UTC

gfv wrote: post #5

cperciva wrote: post #3

blinks Ok, I really wasn't expecting this to land at the top of HN. I'd love to stick around to answer any questions people have, but it's 10PM and my toddler decided to go to bed at 5PM... so if I'm lucky I can get about 4 hours of sleep before she decides that it's time to get up. I'll check in and answer questions in the morning.

How long do you keep the transaction logs before rewriting them? I too had a few EC2 instances go down with signs of being severed from the EBS in the recent couple of weeks; mine were in eu-west.

There's a continual background cleaning process which depends on the amount of storage which can be reclaimed -- there's a tradeoff between cleaning too slowly (and paying for wasted storage) and cleaning too fast (and paying for lots of S3 operations). I think it averages a couple weeks right now.

Re: Tarsnap outage postmortem

#242

Post by andrewmunsell » Thu, Jul 27, 2023, 4:48 PM UTC

ivanhoe wrote: post #159

stigz wrote: post #9

Earlier quoted context omitted.

Why would I use your service over restic? God bless you Colin, but reading this, it appears you're the only one in charge of the infrastructure for this service. I'm glad you're clear about no SLA, but this seems like a big liability between me and my backups.

It's a pretty well-known fact for years that tarsnap is basically a one-man show, and yet Colin has managed to provide fantastic service so far. Sometimes having ppl who built the service also managing it is actually a big plus, compared to other services where you first have to fight through outsourced & underpaid support that's limited to template answers, only to finally get some "engineer" who got that job 2 mont…

I hate to bring this up, but what about the bus factor? If Colin is physically unable to continue maintaining the service and something like this happens again, how will anyone be able to get their data out? It's not really a concern about the service Tarsnap provides today

Re: Tarsnap outage postmortem

#243

Post by rsync » Thu, Jul 27, 2023, 4:48 PM UTC

AnonHP wrote: post #204

mekster wrote: post #29

Earlier quoted context omitted.

It's insane. Not sure how anyone can accept such a rip off pricing. Tarsnap : $0.25 / GB storage, $0.25 / GB bandwidth cost rsync.net : $0.015 / GB storage, no bandwidth cost s3 : $0.023 / GB storage, some complicated bandwidth pricing If tarsnap is built on top of s3, they're charging 10 times for the storage cost. Easy money from the uninformed?

You can get a HN reader’s discount on rsync.net (email them to ask for it or search on HN), bringing the price down to $0.12 / GB, and everything else remains the same.

The HN reader’s discount is lower than that these days (we probably "normalized"[1] parents account to reflect that).

We also have .edu / student / nonprofit discounts. Email us.

Finally, Debian and FreeBSD project members get free accounts. See the committers handbook, etc., for details.

[1] Whenever we lower our prices, we increase quota on existing customers to "normalize" them to the new price/GB. If you do nothing, your rsync.net account just grows over time due to this.

Re: Tarsnap outage postmortem

#244

Post by cperciva » Thu, Jul 27, 2023, 4:49 PM UTC

dharmapure wrote: post #128

cperciva wrote: post #3

blinks Ok, I really wasn't expecting this to land at the top of HN. I'd love to stick around to answer any questions people have, but it's 10PM and my toddler decided to go to bed at 5PM... so if I'm lucky I can get about 4 hours of sleep before she decides that it's time to get up. I'll check in and answer questions in the morning.

Thank you for the post-mortem Colin and I hope you get some sleep!

Thanks, I did! My long suffering wife was up at 3:30 though. :-(

Re: Tarsnap outage postmortem

#245

Post by dang » Thu, Jul 27, 2023, 4:52 PM UTC

switch007 wrote: post #4

Not to be that guy, but it’s unreadable either zoomed in or in reader mode either horizontal or landscape on iOS. Colin, could the website be updated to the 2010s? :P

"Please don't complain about tangential annoyances—e.g. article or website formats, name collisions, or back-button breakage. They're too common to be interesting."

https://news.ycombinator.com/newsguidelines.html

Re: Tarsnap outage postmortem

#246

Post by cperciva » Thu, Jul 27, 2023, 4:53 PM UTC

koolba wrote: post #136

saalweachter wrote: post #126

Earlier quoted context omitted.

I mean, if you are on HN, you will probably learn of a Colin outage within 24 hours, so practically speaking you would really only have a problem if your primary data storage, Tarsnap, and Colin all failed in the same 24 hour window or so before you had time to switch to a new backup provider.

Pretty sure his brother works on tarsnap too. They should take separate buses to ______.

Pretty sure his brother works on tarsnap too.

Yes, I hired him in 2015 IIRC. If you look at tarsnap's GitHub you'll see a lot of commits from gperciva.

Re: Tarsnap outage postmortem

#247

Post by cperciva » Thu, Jul 27, 2023, 4:55 PM UTC

idlewords wrote: post #213

Hats off to you for an honest postmortem and your capable handling of a difficult situation. The only remark I would offer is with respect to sleep deprivation—when you're the only person who can fix a problem, there's no shame in trading some additional outage time for a fresh mind. Though it feels weird to go nap when all the klaxons are blaring, problems are too easy to compound under the combination of adrenaline…

Don't worry, I had a couple naps in there. "This seems to be running smoothly but it will take several more hours; I'll set my alarm to wake me up in two hours and have a nap" is part of why I didn't notice the second step was unnecessarily I/O bound.

Re: Tarsnap outage postmortem

#248

Post by rsync » Thu, Jul 27, 2023, 4:55 PM UTC

deathanatos wrote: post #238

> Following my ill-defined "Tarsnap doesn't have an SLA but I'll give people credits for outages when it seems fair" policy, on 2023-07-13 (after some dust settled and I caught up on some sleep) I credited everyone's Tarsnap accounts with 50% of a month's storage costs. This speaks volumes to me about what kind of person Percival is; that credit would appear to be generously on the "make customer whole" side of the f…

"Thanks for being a beacon of light in the dark."

That's well put.

It makes me very happy to live in a world where tarsnap exists and is priced in picodollars.

Re: Tarsnap outage postmortem

#249

Post by masfuerte » Thu, Jul 27, 2023, 4:58 PM UTC

zetalyrae wrote: post #233

masfuerte wrote: post #231

Earlier quoted context omitted.

The VM crashed, corrupting the file system. This could have made a Postgres database unrecoverable. For rock solid reliability you need more than a database instance.

Make two of them.

Keeping some kind of Postgres cluster running for over a decade seems like a lot of work. tarsnap seems to require roughly no maintenance.

Re: Tarsnap outage postmortem

#250

Post by cperciva » Thu, Jul 27, 2023, 5:00 PM UTC

zokier wrote: post #14

Based on the description it sounds like it should be relatively easy to test this recovery process on a regular basis, to catch any lingering bugs and evaluate the recovery time. As they say, the only backups are the ones you have tested.

Yep! I've been meaning to do it for a while but there was always something higher priority... I didn't realize until this outage that it had been almost a decade since I had tested it.

Rehearsing this annually is definitely going to be a high priority.