Summary of the Amazon S3 Service Disruption • Hacker News Board

Re: Summary of the Amazon S3 Service Disruption

#221

Post by notatoad » Thu, Mar 02, 2017, 6:56 PM UTC

pmoriarty wrote: post #210

mabbo wrote: post #153

Earlier quoted context omitted.

Look at the language used though. This is saying very loudly "Look, this isn't the engineer's fault here". It's one thing I miss about Amazon's culture- not blaming people when system's fail. The follow-up doesn't bullshit with "extra training to make sure no one does this again", it says (effectively) "we're going to make this impossible to happen again, even if someone makes a mistake".

Putting the capability to take down S3 in to the hands of a single engineer seems a bit much. Is mere extra training the right solution here? Maybe they need something like the procedure that's used in missile silos: Not allowing the shutdown system to function at all without the explicit authorization of least two people.

The linked article also says the tools they use were changed to limit the amount of resources that could be taken down at a single time, the speed they could be taken down at, and a hard floor was put on the number of instances that could be stopped.

That's a lot more than just extra training, and a lot better than a two-key system.

Re: Summary of the Amazon S3 Service Disruption

#222

Post by djsumdog » Thu, Mar 02, 2017, 6:56 PM UTC

mabbo wrote: post #153

ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

Look at the language used though. This is saying very loudly "Look, this isn't the engineer's fault here". It's one thing I miss about Amazon's culture- not blaming people when system's fail. The follow-up doesn't bullshit with "extra training to make sure no one does this again", it says (effectively) "we're going to make this impossible to happen again, even if someone makes a mistake".

Which is really the point of automation and configuration management. When a manager asks you, "How are you going to prevent this in the future?" You can say, "We added a check so n must be less than x% of the total number of cluster members," or "We added additional unit tests for the missing area of coverage" or "We added new integration tests that will pick up on this."

Tests and configuration scripts don't prevent all breakage. But when you have them, you can say, "We missed that, let's add it," or "That failed, but it's a false positive. Let's add this edge case to this test."

If you have no automation, tests or auditing systems around running deployments, you can't do any of this.

Re: Summary of the Amazon S3 Service Disruption

#223

Post by notatoad » Thu, Mar 02, 2017, 6:58 PM UTC

idlewords wrote: post #208

andrewaylett wrote: post #191

Earlier quoted context omitted.

Note that's durability not reliability . You might not be able to get at it with every request (I think 99.99% is the target) but it'll still be there if you try again later.

The point is that at eleven nines, you're entering the realm of very rare/unlikely events that will also affect durability. In other words, there's a lack of humility about "unknown unknowns".

but amazon doesn't offer eleven 9s of availability. I don't think anybody serious does, so arguing how silly eleven 9s of availability is is kind of pointless. The SLA is only four 9s of availability.

Re: Summary of the Amazon S3 Service Disruption

#224

Post by aestetix » Thu, Mar 02, 2017, 6:58 PM UTC

It sounds like this can be mitigated by making sure everything is run in dry run mode first, and for something mission critical, getting it double-checked by someone before removing the dry run constraint.

It's good practice in general, and I'm kind of astonished it's not part of the operational procedures in AWS, as this would have quickly been caught and fixed before ever going out to production.

Re: Summary of the Amazon S3 Service Disruption

#225

Post by [deleted] » Thu, Mar 02, 2017, 6:59 PM UTC

kevinchen wrote: post #213

jph00 wrote: post #10

> From the beginning of this event until 11:37AM PST, we were unable to update the individual services’ status on the AWS Service Health Dashboard (SHD) because of a dependency the SHD administration console has on Amazon S3. Ensuring that your status dashboard doesn't depend on the thing it's monitoring is probably the first thing you should think about when designing your status system. This doesn't fill me with co…

I'm curious as to why their fix was to host the Service Health Dashboard on more AWS regions. It seems like the responsible thing to do is to host it entirely on a competitor's service. That way, it's very simple to know that the status page will work no matter what happens to you.

[deleted]

Re: Summary of the Amazon S3 Service Disruption

#226

Post by ghayes » Thu, Mar 02, 2017, 6:59 PM UTC

kevinchen wrote: post #213

jph00 wrote: post #10

> From the beginning of this event until 11:37AM PST, we were unable to update the individual services’ status on the AWS Service Health Dashboard (SHD) because of a dependency the SHD administration console has on Amazon S3. Ensuring that your status dashboard doesn't depend on the thing it's monitoring is probably the first thing you should think about when designing your status system. This doesn't fill me with co…

I'm curious as to why their fix was to host the Service Health Dashboard on more AWS regions. It seems like the responsible thing to do is to host it entirely on a competitor's service. That way, it's very simple to know that the status page will work no matter what happens to you.

Or have it pull from two sources, one local (S3) and one remote (GCE or whatever), and make a hard positive from either source signal "down." Otherwise the page would be down if just the remote source were down.

Re: Summary of the Amazon S3 Service Disruption

#227

Post by liquidise » Thu, Mar 02, 2017, 6:59 PM UTC

mabbo wrote: post #153

ajross wrote: post #8

> At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. It remains amazing to me that even with all the layers of automation, the…

Look at the language used though. This is saying very loudly "Look, this isn't the engineer's fault here". It's one thing I miss about Amazon's culture- not blaming people when system's fail. The follow-up doesn't bullshit with "extra training to make sure no one does this again", it says (effectively) "we're going to make this impossible to happen again, even if someone makes a mistake".

This is the major basis of the CMM Levels [1]. At higher levels of maturity and necessity, systems and processes are designed to increasingly prevent errors from reaching a production environment.

Amazon is taking the right approach here. The fact that a system as complex and important as S3 can be taken down is a failure of the system, not the person who took it down accidentally.

1. https://en.wikipedia.org/wiki/Capability_Maturity_Model#Leve...

Re: Summary of the Amazon S3 Service Disruption

#228

Post by djsumdog » Thu, Mar 02, 2017, 7:00 PM UTC

kevinchen wrote: post #213

jph00 wrote: post #10

> From the beginning of this event until 11:37AM PST, we were unable to update the individual services’ status on the AWS Service Health Dashboard (SHD) because of a dependency the SHD administration console has on Amazon S3. Ensuring that your status dashboard doesn't depend on the thing it's monitoring is probably the first thing you should think about when designing your status system. This doesn't fill me with co…

I'm curious as to why their fix was to host the Service Health Dashboard on more AWS regions. It seems like the responsible thing to do is to host it entirely on a competitor's service. That way, it's very simple to know that the status page will work no matter what happens to you.

You have to eat your own dogfood, if you're trying to sell a service product.

That being said, a better solution would be to stand up a separate infrastructure just for status pages.

Re: Summary of the Amazon S3 Service Disruption

#229

Post by notatoad » Thu, Mar 02, 2017, 7:00 PM UTC

losteverything wrote: post #104

"I did." That was CEO Robert Allen's response when the AT&T network collapsed [1] on January 15, 1990 He was asked who made the mistake. I can't imagine any CEO now a days making a similar statement. [1] http://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collaps...

I'd argue that the message of this post-mortem, that is "mistakes were made, but the fault is with the tools and not any one person" is a much better response than the CEO making a symbolic statement claiming the fault.

both better for morale, and better for preventing another incident.

Re: Summary of the Amazon S3 Service Disruption

#230

Post by pmoriarty » Thu, Mar 02, 2017, 7:00 PM UTC

kevinchen wrote: post #213

jph00 wrote: post #10

> From the beginning of this event until 11:37AM PST, we were unable to update the individual services’ status on the AWS Service Health Dashboard (SHD) because of a dependency the SHD administration console has on Amazon S3. Ensuring that your status dashboard doesn't depend on the thing it's monitoring is probably the first thing you should think about when designing your status system. This doesn't fill me with co…

I'm curious as to why their fix was to host the Service Health Dashboard on more AWS regions. It seems like the responsible thing to do is to host it entirely on a competitor's service. That way, it's very simple to know that the status page will work no matter what happens to you.

If they did host their status page on a competitor's service, then they'd be reliant on that service, which might backfire if the competitor's service goes down while Amazon's own systems stay up.

What they really need is failover capability, which can fire up the status page on a competitor's service (or maybe on a completely separate disaster recovery site site owned by Amazon) in case Amazon's own services go down.

I'm sure Amazon's architects and engineers are more than capable of designing and implementing such a robust system and recognizing its importance. So it puzzles me as to why it wasn't done.