Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS Service Disruption Post Mortem

aws.amazon.com

Post reply on HN
Print view View original

71–80 of 108 posts

Re: AWS Service Disruption Post Mortem

#71

Post by brown9-2 »

hobbes wrote: post #6

>...one of the standard steps is to shift traffic off of one of the redundant routers in the primary EBS network to allow the upgrade to happen. The traffic shift was executed incorrectly... This supports the theory that between 50%-80% of outages are caused by human error, regardless of the resilience of the underlying infrastructure.

This supports the theory that between 50%-80% of outages are caused by human error

Not quite - in this case, a single human error then triggered a series of latent and undiscovered bugs in the system itself. It's a confluence of small events that makes for a large-scale problem like this.

Re: AWS Service Disruption Post Mortem

#72

Post by ssmoot »

lemming wrote: post #57
jwe wrote: post #42

Earlier quoted context omitted.

It's not so much about thinking that somebody else could do better. By doing what they do they create the expectation that they _are_ doing better than everybody else.

And let's face it, despite bugs and human error that should be avoidable, they are.

At that scale, MAYBE.

In the last two years my workplace has gotten pretty good at handling SAN outages (due to terrible Oracle equipment).

Put simply, this set of scenarios can't happen at my work-place. We don't have that level of automation, there's only a pair of SAN systems in the mirrors, there's no "hunting for capacity".

I'd suggest most businesses are closer to this than AWS.

There's a lot of sugar coating going around saying "You couldn't build a better space-shuttle", and that's probably true. But if I only need an extremely reliable bicycle that's a false argument. The Simplest Thing That Could Possibly Work doesn't apply just to programming.

Re: AWS Service Disruption Post Mortem

#73

Post by space-monkey »

ra wrote: post #65
extension wrote: post #63

Earlier quoted context omitted.

I don't see how transparently replicated storage could be implemented without sharing something .

In which case they shouldn't replicate across availability zones. 'shared nothing' is the only way to islandize failures.

And that's why they have multiple fully-isolated regions. Availability zones are a purposeful tradeoff that provides easier to use service with higher inter-zone communication performance and lower cost.

Re: AWS Service Disruption Post Mortem

#74

Post by krobertson »

mml wrote: post #26

Did I read this correctly in paragraph 2: " For two periods during the first day of the issue, the degraded EBS cluster affected the EBS APIs and caused high error rates and latencies for EBS calls to these APIs across the entire US East Region." Their "control plane" network for the EBS clusters span availability zones in a region? If so, this would be the fatal flaw.

To me, it sounds like a large single point of failure, and the post-mortem doesn't seem to acknowledge that or discuss remedying it.

They setup a separate instance of it to help with API calls in the affected region, but it still sounds like it functions across AZs and is still vulnerable overall.

Re: AWS Service Disruption Post Mortem

#75

Post by gruseom »

wanderr wrote: post #52

I highly recommend that anyone who was surprised by this outage, or the description of the chain reaction of failures that lead to it, read Systemantics. It is a dry but amusing exploration of the seemingly universal fact that every complex system is always operating in a state of failure, but the complexity, failovers and multiple layers can hide this, until the last link in the chain finally breaks, usually with ca…

read Systemantics

Oh yes. It's a classic that deserves to be much better known. Anybody engaged with complex systems - such as software or software projects - will find all kinds of suggestive things in there. As for "dry"... come now, it's hilarious and has cartoons.

Basically, just get it. Here, I'll help:

http://www.amazon.com/Systems-Bible-Beginners-Guide-Large/dp...

(They ruined the title but it's the same book.)

Re: AWS Service Disruption Post Mortem

#76

Post by pwzeus »

I for once just want to say that claps to them for figuring this out , nailing it down in fixing it in just few days. After reading this if feels like issue at such massive level can take large amount of time to fix.

Re: AWS Service Disruption Post Mortem

#77

Post by SoftwareMaven »

ekidd wrote: post #50
nicpottier wrote: post #17

tldr: ""The trigger for this event was a network configuration change. We will audit our change process and increase the automation to prevent this mistake from happening in the future." AMZN has gotten a lot of flack over this outage, and rightly so. But I do want to dissuade anyone from thinking anybody else could do much better. I worked there 10 years ago, when they were closer to 200 engineers, and the caliber o…

Given a choice between hosting servers on AWS, and trying to build my own reliable infrastructure with a single sysadmin, I'll take AWS in a heartbeat. But I do want to quibble with one of your points: It is easy to be an arm chair coach and scream that... they should have predicted this storm I'm not as smart as the AWS developers, and I have a lot less experience with large-scale distributed systems. But thanks to…

I would be surprised if Amazon didn't do testing similar to the Google quote above. AWS is insanely complicated, and the problem with an insanely complicated system is there are infinite failure modes (or such a big number it is close enough to infinite for human purposes). As such, it is impossible to test all of them.

Re: AWS Service Disruption Post Mortem

#78

Post by icedpulleys »

nicpottier wrote: post #17

tldr: ""The trigger for this event was a network configuration change. We will audit our change process and increase the automation to prevent this mistake from happening in the future." AMZN has gotten a lot of flack over this outage, and rightly so. But I do want to dissuade anyone from thinking anybody else could do much better. I worked there 10 years ago, when they were closer to 200 engineers, and the caliber o…

My tl;dr reads differently: "we lacked appropriate congestion control in our EBS recovery algorithms."

The trigger was a bad and unexpected network configuration change, but the error was that the attempted recovery by the stuck volumes was that it was uncontrolled.

I don't think that anyone is knocking the intelligence of the AWS engineers, or saying that anyone else could do it better. Just like NASA engineers and scientists are incredibly intelligent and good at what they do, systems can become complicated enough that unexpected errors creep into the system, and not any particular component.

Re: AWS Service Disruption Post Mortem

#80

Post by eddieplan9 »

ekidd wrote: post #50
nicpottier wrote: post #17

tldr: ""The trigger for this event was a network configuration change. We will audit our change process and increase the automation to prevent this mistake from happening in the future." AMZN has gotten a lot of flack over this outage, and rightly so. But I do want to dissuade anyone from thinking anybody else could do much better. I worked there 10 years ago, when they were closer to 200 engineers, and the caliber o…

Given a choice between hosting servers on AWS, and trying to build my own reliable infrastructure with a single sysadmin, I'll take AWS in a heartbeat. But I do want to quibble with one of your points: It is easy to be an arm chair coach and scream that... they should have predicted this storm I'm not as smart as the AWS developers, and I have a lot less experience with large-scale distributed systems. But thanks to…

But thanks to my own cluelessness, I've blown up smaller distributed systems, and I've learned one important lesson: Almost nobody is smart enough to understand automatic error-recovery code. Features like automated volume remirroring or multi-AZ failover increase the load on an already stressed system, and they often cause this kind of "storm."

It's basically Test-Driven Development: if you cannot test it, don't write it.

Post reply on HN