AWS Service Disruption Post Mortem • Hacker News Board

Re: AWS Service Disruption Post Mortem

#101

Post by space-monkey » Sat, Apr 30, 2011, 3:14 AM UTC

ra wrote: post #99

space-monkey wrote: post #73

Earlier quoted context omitted.

And that's why they have multiple fully-isolated regions. Availability zones are a purposeful tradeoff that provides easier to use service with higher inter-zone communication performance and lower cost.

No, that's not what Amazon says. The following is from the AWS web site [1]: > Availability Zones are distinct locations that are engineered to be insulated from failures in other Availability Zones and provide inexpensive, low latency network connectivity to other Availability Zones in the same Region. By launching instances in separate Availability Zones, you can protect your applications from failure of a single l…

The tradeoff is in that AZs are "engineered to be insulated" as opposed to being actually, or naturally isolated. Prior to their downtime, I've had plenty of conversations with folks that I work with about AWS and we've always assumed that AZs are not 100% isolated. I can see how someone can read "engineered to be insulated" the other way, but I generally read these kinds of materials as guaranteeing nothing beyond the most limited possible reading, and probably not even that.

The quoted statement doesn't say that isolation is 100% or that multiple AZs can't ever ever fail at the same time. It says that if only one AZ goes down and you have servers in another, then those servers will still be up, which should be obvious. Insulated doesn't even mean the same thing as isolated.

Re: AWS Service Disruption Post Mortem

#102

Post by john61 » Sat, Apr 30, 2011, 5:22 AM UTC

bretthopper wrote: post #67

I've been noticing a trend recently when reading about large scale failures of any system: it's never just one thing. AWS EBS outage, Fukushima, Chernobyl, even the great Chicago Fire (forgive me for comparing AWS to those events). Sure there's always a "root" cause, but more importantly, it's the related events that keep adding up to make the failure even worse. I can only imagine how many minor failures happen worl…

That's why Sergei Korolev the chief engineer behind the successfull russion space program that led to Gargarins victory said:

„The genius of a construction lies in its simplicity. Everybody can build complicated things."

Re: AWS Service Disruption Post Mortem

#103

Post by ra » Sat, Apr 30, 2011, 10:57 AM UTC

space-monkey wrote: post #101

ra wrote: post #99

Earlier quoted context omitted.

No, that's not what Amazon says. The following is from the AWS web site [1]: > Availability Zones are distinct locations that are engineered to be insulated from failures in other Availability Zones and provide inexpensive, low latency network connectivity to other Availability Zones in the same Region. By launching instances in separate Availability Zones, you can protect your applications from failure of a single l…

The tradeoff is in that AZs are "engineered to be insulated" as opposed to being actually, or naturally isolated. Prior to their downtime, I've had plenty of conversations with folks that I work with about AWS and we've always assumed that AZs are not 100% isolated. I can see how someone can read "engineered to be insulated" the other way, but I generally read these kinds of materials as guaranteeing nothing beyond t…

Sure, I don't disagree with what you are saying. However I think that the way Amazon presents the concept of an AZ is that it IS isolated from other AZ's in the same region.

Even the name, 'Availability Zone' implies that it is isolated from other 'Availability Zones' in the same region. And that text I quoted does nothing but substantiate that inference.

I just think that Amazon are misleading here. Maybe they shouldn't call it an Availability Zone.

Re: AWS Service Disruption Post Mortem

#104

Post by OllieJones » Sat, Apr 30, 2011, 11:15 AM UTC

ianso wrote: post #53

nicpottier wrote: post #17

tldr: ""The trigger for this event was a network configuration change. We will audit our change process and increase the automation to prevent this mistake from happening in the future." AMZN has gotten a lot of flack over this outage, and rightly so. But I do want to dissuade anyone from thinking anybody else could do much better. I worked there 10 years ago, when they were closer to 200 engineers, and the caliber o…

Agreed with the expertise thing, but to add to your summary of the post-mortem, it looks like human error compounded by: 1) An architecture bug (the EBS "control plane" cuts across Availability Zones and EBS clusters, leading to a single point of failure: this is what broke the "service contract"), 2) a spec/programming bug: No aggressive back-off on retry attempts of EBS ops, and 3) two separate logical bugs: the ra…

Agree completely. We are still in the early days of AWS-like technology. Providers and users both will experience some serious issues in the years to come.

Heck, users of municipal water systems still experience outages and that technology is arguably mature.

The really encouraging thing is that the Amazon post-mortem writeup indicates they're taking it very seriously.

Re: AWS Service Disruption Post Mortem

#105

Post by JohnsonB » Sat, Apr 30, 2011, 7:38 PM UTC

nicpottier wrote: post #17

tldr: ""The trigger for this event was a network configuration change. We will audit our change process and increase the automation to prevent this mistake from happening in the future." AMZN has gotten a lot of flack over this outage, and rightly so. But I do want to dissuade anyone from thinking anybody else could do much better. I worked there 10 years ago, when they were closer to 200 engineers, and the caliber o…

A single manual configuration mistake should never be able cause this type of complete system failure. For AWS to be engineered this way, the engineers you met must not have been as smart as they seemed. EBS may be extremely sophisticated in other respects, but if this type of thing was even a possibility, the AWS team are very fallible, and probably far from the best in their field.

>In short, take my word for it, the people working on this are smarter than you and me, by an order of magnitude. There is no way you could do better, and it is unlikely that if you are building anything that needs more than a handful of servers you could build anything more reliable.

Ever since the AWS outage, I've seen a number of these "the AWS guys are so smart, I've met them." type comments. And then paraphrased: "There sort-of can't be that much to blame on them because of how smart they are, and they are so smart anyway, who could do better?" That's not a valid argument, not everyone is equally impressed by an individuals intelligence, perhaps your assessment is wrong. And even if someone is insanely smart, they still can commit practical errors which indicates they are smart, but still flawed in their understanding of engineering in significant ways. Perhaps AWS simply does need a higher caliber of engineer that wouldn't miss out on these dead-simple safeguards that would have prevented this outage.

Re: AWS Service Disruption Post Mortem

#106

Post by evangineer » Sat, Apr 30, 2011, 8:17 PM UTC

jsdalton wrote: post #59

mml wrote: post #26

Did I read this correctly in paragraph 2: " For two periods during the first day of the issue, the degraded EBS cluster affected the EBS APIs and caused high error rates and latencies for EBS calls to these APIs across the entire US East Region." Their "control plane" network for the EBS clusters span availability zones in a region? If so, this would be the fatal flaw.

I may have read it incorrectly myself, but I interpreted this as meaning the control plane was balanced across availability zones in order to provide durability in the face of a failure of one of the zones. In other words, Amazon is ensured their control plane is operational at all times. The API failures were ultimately tied to the network problems that occurred, not to a failure of the control plane. EDIT: I should…

They seem to have settled on a halfway house, pushing more of the control plane functionality down into the EBS clusters and making the remaining shared control plane more robust to the sort of failures that arose this time.

Re: AWS Service Disruption Post Mortem

#107

Post by olefoo » Sat, Apr 30, 2011, 9:28 PM UTC

bretthopper wrote: post #67

I've been noticing a trend recently when reading about large scale failures of any system: it's never just one thing. AWS EBS outage, Fukushima, Chernobyl, even the great Chicago Fire (forgive me for comparing AWS to those events). Sure there's always a "root" cause, but more importantly, it's the related events that keep adding up to make the failure even worse. I can only imagine how many minor failures happen worl…

A related book that examines some of the same themes is Charles Perrow's Normal Accidents http://press.princeton.edu/titles/6596.html

The examples he draws from are nuclear power plant failures (TMI in particular), civil aviation and oil transport. But the basics will be recognizable to anyone who has dealt with large computing installations; interactive complexity, tight coupling and cascading failures.

It is not a reassuring book, you won't be able to look at any complex system without asking yourself what sequence of simple, predictable failures of widely separated parts could tip it into a catastrophic failure mode.

Re: AWS Service Disruption Post Mortem

#108

Post by shaloum » Sun, May 01, 2011, 2:18 PM UTC

ekidd wrote: post #50

nicpottier wrote: post #17

tldr: ""The trigger for this event was a network configuration change. We will audit our change process and increase the automation to prevent this mistake from happening in the future." AMZN has gotten a lot of flack over this outage, and rightly so. But I do want to dissuade anyone from thinking anybody else could do much better. I worked there 10 years ago, when they were closer to 200 engineers, and the caliber o…

Given a choice between hosting servers on AWS, and trying to build my own reliable infrastructure with a single sysadmin, I'll take AWS in a heartbeat. But I do want to quibble with one of your points: It is easy to be an arm chair coach and scream that... they should have predicted this storm I'm not as smart as the AWS developers, and I have a lot less experience with large-scale distributed systems. But thanks to…

No one seems smart enough though ;) http://groups.google.com/group/google-appengine-downtime-not...