Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS Service Disruption Post Mortem

aws.amazon.com

Post reply on HN
Print view View original

51–60 of 108 posts

Re: AWS Service Disruption Post Mortem

#51

Post by wanderr »

RyanKearney wrote: post #23
tomjen3 wrote: post #8

Earlier quoted context omitted.

Well yes, except that that is usually peanuts compared to the lost income from your service being down. Really the only purpose of a SLA penalty is to incentivize the provider to keep the network reliable.

I still don't know why people keep throwing this out there. Yes, if you were effected and it caused you to lose a lot of income because your business could not operate when EC2 was on the fritz then it's your fault, not Amazons. You can't keep blaming Amazon because you didn't build a fault-tolerant application. It's like blaming your electric company because your home is lit by one super huge flood light which burnt…

I think it's more like getting upset with your cable provider when their service goes down leaving you stranded with no internet. Arguably, it's your fault that you don't have any internet, you could have had redundant internet access between cable and DSL and you have no one else to blame for you decision to pay for only one.

Re: AWS Service Disruption Post Mortem

#52

Post by wanderr »

I highly recommend that anyone who was surprised by this outage, or the description of the chain reaction of failures that lead to it, read Systemantics. It is a dry but amusing exploration of the seemingly universal fact that every complex system is always operating in a state of failure, but the complexity, failovers and multiple layers can hide this, until the last link in the chain finally breaks, usually with catastrophic results.

Re: AWS Service Disruption Post Mortem

#53

Post by ianso »

nicpottier wrote: post #17

tldr: ""The trigger for this event was a network configuration change. We will audit our change process and increase the automation to prevent this mistake from happening in the future." AMZN has gotten a lot of flack over this outage, and rightly so. But I do want to dissuade anyone from thinking anybody else could do much better. I worked there 10 years ago, when they were closer to 200 engineers, and the caliber o…

Agreed with the expertise thing, but to add to your summary of the post-mortem, it looks like human error compounded by: 1) An architecture bug (the EBS "control plane" cuts across Availability Zones and EBS clusters, leading to a single point of failure: this is what broke the "service contract"), 2) a spec/programming bug: No aggressive back-off on retry attempts of EBS ops, and 3) two separate logical bugs: the race condition in the EBS nodes & problems with MySQL replication.

I think that's everything. It just goes to show, most disasters in very well-engineered systems are generally the result of a series of things all going wrong at once, not individual failures...

Re: AWS Service Disruption Post Mortem

#54

Post by gord »

This article reads like nonsense - but this is not a criticism of AWS.

The real problem is there is no good mathematical model of distributed behaviour, from which statistical guarantees can be made.

I think we're at the limit of what the smartest people can achieve with hand crafted code.

Most likely new math will give rise to new tools and languages, in which the next generation of reliable distributed systems will be written.

Without this advance we will have storage networks that aren't reliable, an internet that can be taken down by one organization, botnets that are unkillable and patchy network security.

Re: AWS Service Disruption Post Mortem

#55

Post by cosmicray »

com wrote: post #24
tomjen3 wrote: post #8

Earlier quoted context omitted.

Well yes, except that that is usually peanuts compared to the lost income from your service being down. Really the only purpose of a SLA penalty is to incentivize the provider to keep the network reliable.

I totally agree with your comment about the SLA penalty as an incentive to the provider to take reasonable measures to ensure service. But that's just in general. When negotiating bespoke SLA penalty clauses, it can be very illuminating for both sides to discuss lost profit + lost confidence + additional costs to the customer and suggest that these be factored in to the penalty clause. My experience: both the custome…

> In a off-the-shelf service like Amazon, you as a customer are welcome to suggest a change of penalty to your Amazon account manager, and unless you're something like the US government, you will probably be directed to other cloud providers or your own internal IT organisation!

What that suggests to me, is that the time has arrived for an external organization, one that sells loss-of-business protection against such failures, needs to become involved. Such an organization, should enough cloud customers subscribe to it, would become an influence upon services like AWS. I'm not sure I 'like' this idea, but the premise that a customer is using the cloud service at the whim of whatever the provider decides is best practice needs to be revisited.

Re: AWS Service Disruption Post Mortem

#57

Post by lemming »

jwe wrote: post #42
nicpottier wrote: post #17

tldr: ""The trigger for this event was a network configuration change. We will audit our change process and increase the automation to prevent this mistake from happening in the future." AMZN has gotten a lot of flack over this outage, and rightly so. But I do want to dissuade anyone from thinking anybody else could do much better. I worked there 10 years ago, when they were closer to 200 engineers, and the caliber o…

It's not so much about thinking that somebody else could do better. By doing what they do they create the expectation that they _are_ doing better than everybody else.

And let's face it, despite bugs and human error that should be avoidable, they are.

Re: AWS Service Disruption Post Mortem

#58

Post by rlpb »

nicpottier wrote: post #17

tldr: ""The trigger for this event was a network configuration change. We will audit our change process and increase the automation to prevent this mistake from happening in the future." AMZN has gotten a lot of flack over this outage, and rightly so. But I do want to dissuade anyone from thinking anybody else could do much better. I worked there 10 years ago, when they were closer to 200 engineers, and the caliber o…

> There is no way you could do better, and it is unlikely that if you are building anything that needs more than a handful of servers you could build anything more reliable.

I can't disagree, but there is one key benefit to not using the cloud for some services.

When your company is working on an important deadline, your sysadmins could choose not to implement that pending network configuration change during that crucial period. You can control your own at-risk times, which you can't generally do with IaaS.

As with everything, it's a trade off.

Re: AWS Service Disruption Post Mortem

#59

Post by jsdalton »

mml wrote: post #26

Did I read this correctly in paragraph 2: " For two periods during the first day of the issue, the degraded EBS cluster affected the EBS APIs and caused high error rates and latencies for EBS calls to these APIs across the entire US East Region." Their "control plane" network for the EBS clusters span availability zones in a region? If so, this would be the fatal flaw.

I may have read it incorrectly myself, but I interpreted this as meaning the control plane was balanced across availability zones in order to provide durability in the face of a failure of one of the zones. In other words, Amazon is ensured their control plane is operational at all times.

The API failures were ultimately tied to the network problems that occurred, not to a failure of the control plane.

EDIT: I should finish reading before I reply. :) It would appear that the network issue in the one availability zone was so severe that the control plane ran out of threads to service API requests to any of the availability zones.

So while it's true the underlying problem was a network issue, the fact that the the control plane is spread across availability zones was responsible for part of the outage that occurred across the whole region.

My totally unqualified assessment of this aspect of the outage is that, while it might make sense to have a control plane spread across availability zones, they presumably need to have isolated control planes for each zone, instead of a shared plane as they seemingly have now.

Re: AWS Service Disruption Post Mortem

#60

Post by rdl »

I still don't see a good justification for keeping the ebs control plane exposed to failure across multiple availability zones in a region. Until that is fixed, I would not depend on AZs for real fault tolerance.
Post reply on HN