Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tarsnap outage postmortem

mail.tarsnap.com

Post reply on HN
Print view View original

211–220 of 338 posts

Re: Tarsnap outage postmortem

#211

Post by jrockway »

switch007 wrote: post #53
TheDong wrote: post #10

Earlier quoted context omitted.

It's not Colin's fault that you're using a browser that can't render an html rendition of an email which has been widely in use since before iOS existed. This is entirely Safari's fault for not having good compatibility with a common existing webpage format. Anyway, if you're the intended audience (someone using tarsnap), you also received a copy to your email address, where you can read the text with your email read…

A is the correct kind of HTML use for the body of plain text email ? It looks like paragraphs of text to me. is far more appropriate That isn’t apple’s problem, nor mine.

 is the correct HTML.  People writing plain-text email expect to be able to do things like add ASCII diagrams:
  -------        -------
  | foo |  --->  | bar |
  -------        -------


It's an older technology but it checks out.

Re: Tarsnap outage postmortem

#212

Post by foldr »

zetalyrae wrote: post #197
foldr wrote: post #191

Earlier quoted context omitted.

That sort of logic doesn't really apply here because: * RDS costs obviously scale linearly with ongoing time and probably scale linearly with the total amount of data being backed up. So depending on the revenue of the business, these extra costs could easily end up outweighing the (notional) cost of the time saved, which is mostly a one-off expense. * The cost of a software engineer's time is notional in the context…

[flagged]

The outage was caused by a hardware failure and (I assume) the lack of any redundancy. Using RDS wouldn't have made a difference as far as I can see.

Re: Tarsnap outage postmortem

#213

Post by idlewords »

Hats off to you for an honest postmortem and your capable handling of a difficult situation. The only remark I would offer is with respect to sleep deprivation—when you're the only person who can fix a problem, there's no shame in trading some additional outage time for a fresh mind. Though it feels weird to go nap when all the klaxons are blaring, problems are too easy to compound under the combination of adrenaline and inadequate sleep.

Re: Tarsnap outage postmortem

#214

Post by zetalyrae »

foldr wrote: post #212
zetalyrae wrote: post #197

Earlier quoted context omitted.

[flagged]

The outage was caused by a hardware failure and (I assume) the lack of any redundancy. Using RDS wouldn't have made a difference as far as I can see.

RDS can have replication.

But more than that: servers should be stateless! A server going down should never take down your business.

If you use Postgres, and stateless servers, then if a server goes down it's no problem, it gets rebooted and there may be other servers and a load balancer to pick up the load. If Postgres goes down, you have a replica, or it gets rebooted, and Postgres always recovers from crashes (in my experience), and if it doesn't you have PITR.

AWS has everything under the sun to prevent this kind of thing happening. This is a 1990's outage. This didn't have to happen.

Re: Tarsnap outage postmortem

#215

Post by idlewords »

cowsandmilk wrote: post #158
deltarholamda wrote: post #156

Earlier quoted context omitted.

While I get this as a risk, I'm not convinced it's any more risky than a larger corporate entity. This is Colin's job. Colin has his name attached to it. It's really important to Colin. You're not going to get the same kind of service from BigBackupCorp. Their employees are replaceable, their management is replaceable, and to be honest, you as a customer are replaceable, if they decide to move in a different directio…

It isn’t necessarily about Colin quitting. Key person gets hit by bus is also always a concern. You can say someone will pick it up, but I know nothing of whether such plans are in place. Does the person who would inherit the business have the know how to sell it? Is there enough documentation in place for a transfer of assets to be successful?

This is how that scenario shakes out:

  1. Key person gets hit by bus
  2. You see the black bar on Hacker News and learn the sad news
  3. You go download all your data from the service, which is still up because there is no bus access to data centers.
  4. You feel like a jerk for all your creepy "hit by bus" talk.
  5. A few weeks later, some VC-funded operation with multiple employees you depended on disappears overnight without a trace.

Re: Tarsnap outage postmortem

#216

Post by nodesocket »

cperciva wrote: post #3

blinks Ok, I really wasn't expecting this to land at the top of HN. I'd love to stick around to answer any questions people have, but it's 10PM and my toddler decided to go to bed at 5PM... so if I'm lucky I can get about 4 hours of sleep before she decides that it's time to get up. I'll check in and answer questions in the morning.

Some recommendations on the AWS front (not sure if some of these are already implemented since the postmortem does not go into AWS details).

- Setup nightly automatic snapshots of EBS volumes (this is supported natively now in AWS under lifecycle manager).

- Use EBS volumes of the new GP3 type, and perhaps use provisioned IOPS.

- Setup a auto-scaling group with automatic failover. Of course increases cost, but should be able to automatically failover to a standby EC2 instance (assuming all the code works automatically which the blog post indicates is not currently the case).

Re: Tarsnap outage postmortem

#217

Post by foldr »

zetalyrae wrote: post #214
foldr wrote: post #212

Earlier quoted context omitted.

The outage was caused by a hardware failure and (I assume) the lack of any redundancy. Using RDS wouldn't have made a difference as far as I can see.

RDS can have replication. But more than that: servers should be stateless! A server going down should never take down your business. If you use Postgres, and stateless servers, then if a server goes down it's no problem, it gets rebooted and there may be other servers and a load balancer to pick up the load. If Postgres goes down, you have a replica, or it gets rebooted, and Postgres always recovers from crashes (in…

The hardware failure was on the server running the application code, so RDS replication wouldn’t have helped. You’re right of course that this failure points to a lack of redundancy – but that’s a separate issue from choosing S3 vs. RDS as the data layer.

By the way, S3 is insanely reliable and in fact more reliable than a replicated RDS setup. So switching from S3 to RDS would almost certainly reduce the basic reliability of the data layer, however many conveniences it might bring.

Re: Tarsnap outage postmortem

#218

Post by bastawhiz »

zetalyrae wrote: post #153
bastawhiz wrote: post #150

Earlier quoted context omitted.

> having to reimplement all the consistency, atomicity, transactions etc. on top. Most of those problems are moot if you're only ever writing from a single head node. If all your data is strictly ordered and you have no meaningful concurrency, this is a far, far simpler system.

Did I fall into a timewarp into the 70's? How on Earth, by what sane standard, is a Postgres instance too complex? If you're `fopen`ing files as a "database" you are wasting your time and lowering the world's economic productivity. Complex is Greenspunning a database and having it blow up in your face and cause a twenty-six hour outage. You never hear about such things with Postgres because Postgres is rock-solid.

I'm not defending the choice to use S3. I probably wouldn't have made the same choice. But I am pointing out that it's empirically wrong to say that storing data in flat files necessitates the considerations of an ACID-capable RDBMS.

But to your point, if your system requires less than a thousand lines of code to open a file, do basic parsing and processing (which no data storage system is going to do anyway), and write the output to another file, I personally can't say that Postgres or MySQL or any other solution is really worth the effort/cost to build and maintain. In the system being discussed, the benefits of an RDBMS simply don't matter: any strongly consistent key-value store would work.

> Complex is Greenspunning a database and having it blow up in your face and cause a twenty-six hour outage.

S3 didn't cause the outage, and from the look of it, neither did the code that processes the files. It was an application logic problem which caused issues during the restore process, and this would have been an issue regardless.

You could make an argument that the recovery being slower than it could have been was a problem, but it's wild to say that and imply that traditional databases have no performance cliffs. Especially when dealing with corruption or data recovery. Raw file storage will never have a Postgres transaction id wraparound incident (see: Sentry outage for most of a day in 2015, MailChimp/Mandrill for over a day in 2019) or have to rebuild a critical index.

In this case there was a hard coded concurrency limit with S3 of 250 outstanding requests. Bumping that up to 3000 would have been easy and reasonable (S3 rate limits at 5000). How confident would you be that your database can performantly handle a backfill during recovery? Have you provisioned enough iops? Are you running an RDS instance with only a burstable vCPU limit? To say Postgres is "rock-solid" (and make no mistake, I am a Postgres fanboy) dismisses the many and varying ways that it can fail in unusual and surprising ways.

Re: Tarsnap outage postmortem

#219

Post by catiopatio »

zetalyrae wrote: post #214
foldr wrote: post #212

Earlier quoted context omitted.

The outage was caused by a hardware failure and (I assume) the lack of any redundancy. Using RDS wouldn't have made a difference as far as I can see.

RDS can have replication. But more than that: servers should be stateless! A server going down should never take down your business. If you use Postgres, and stateless servers, then if a server goes down it's no problem, it gets rebooted and there may be other servers and a load balancer to pick up the load. If Postgres goes down, you have a replica, or it gets rebooted, and Postgres always recovers from crashes (in…

Just … stop. This is the first outage in 11 years.

You’re being unnecessarily arrogant and antagonistic up and down this thread

You’re not smarter than everyone else here, you don’t have better or more perfect knowledge, and you almost certainly wouldn’t have built a better or more reliable system.

Re: Tarsnap outage postmortem

#220

Post by idlewords »

jacquesm wrote: post #192
cperciva wrote: post #3

blinks Ok, I really wasn't expecting this to land at the top of HN. I'd love to stick around to answer any questions people have, but it's 10PM and my toddler decided to go to bed at 5PM... so if I'm lucky I can get about 4 hours of sleep before she decides that it's time to get up. I'll check in and answer questions in the morning.

In future postmortems (of which I hope there will be very few or even none) you may want to spell out your 'lessons learned' to show why particular items will never recur.

It always amuses me how people want reassurance that the next crisis will be a fresh, new problem, and not one the person can demonstrably solve.

A lot of 'lessons learned' analysis boils down to this: in order to prevent a recurrence of X, we introduced complex subsystem Y, the unexpected effects of which you can read about in our next post-mortem.

Post reply on HN