Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS Service Disruption Post Mortem

aws.amazon.com

Post reply on HN
Print view View original

41–50 of 108 posts

Re: AWS Service Disruption Post Mortem

#41

Post by thebootstrapper »

biot wrote: post #34
thebootstrapper wrote: post #21

One of the main cause for "re-mirroring storm," is node not backing off from finding a replica. Here's Twitter Back off decider implementation (Java) https://github.com/twitter/commons/blob/master/src/java/com/... When last time i looked i was little clueless on this. Now I find its usage.

Actual URL with Libya dependency removed: https://github.com/twitter/commons/blob/master/src/java/com/... HN doesn't have a 140 character limit, so there's no need to post an obfuscated shortened link.

My fault. Edited the link. Thanks.

Re: AWS Service Disruption Post Mortem

#42

Post by jwe »

nicpottier wrote: post #17

tldr: ""The trigger for this event was a network configuration change. We will audit our change process and increase the automation to prevent this mistake from happening in the future." AMZN has gotten a lot of flack over this outage, and rightly so. But I do want to dissuade anyone from thinking anybody else could do much better. I worked there 10 years ago, when they were closer to 200 engineers, and the caliber o…

It's not so much about thinking that somebody else could do better.

By doing what they do they create the expectation that they _are_ doing better than everybody else.

Re: AWS Service Disruption Post Mortem

#43

Post by joelhaasnoot »

mml wrote: post #26

Did I read this correctly in paragraph 2: " For two periods during the first day of the issue, the degraded EBS cluster affected the EBS APIs and caused high error rates and latencies for EBS calls to these APIs across the entire US East Region." Their "control plane" network for the EBS clusters span availability zones in a region? If so, this would be the fatal flaw.

Isn't this "flaw" a balance of features vs reliability? In most cases it's ok that the API spans the entire region, and makes it easy to address one API endpoint per region, allows Amazon to offer everyone different availibility zones within the region, etc.

Re: AWS Service Disruption Post Mortem

#44

Post by kamaal »

nicpottier wrote: post #17

tldr: ""The trigger for this event was a network configuration change. We will audit our change process and increase the automation to prevent this mistake from happening in the future." AMZN has gotten a lot of flack over this outage, and rightly so. But I do want to dissuade anyone from thinking anybody else could do much better. I worked there 10 years ago, when they were closer to 200 engineers, and the caliber o…

The best are human beings too! I mean even the best work in two ways. Build on best principles already known, and make your own best principles. The first depends on how much you know, how much time you have taken to learn, implement, practice and perfect. The second depends on how far you can see.

In either case you can always overlook or fail to predict the even the easily foreseeable future. And that happens due to many reasons plain human error, or even over confidence which is some times the case with the best.

The best way out of this problem is what Jeff Atwood had blogged some time days back. Is to keep failing, and keep failing in different ways. And each failure needs to be translated into to lessons of some sort and then the solution of it a best practice. Even the Netflix model of failing purposefully will do.

There is no way the best can be flawless. Nothing is flawless, so as long it is done by a human.

Re: AWS Service Disruption Post Mortem

#45

Post by nicpottier »

fauigerzigerk wrote: post #35
nicpottier wrote: post #17

tldr: ""The trigger for this event was a network configuration change. We will audit our change process and increase the automation to prevent this mistake from happening in the future." AMZN has gotten a lot of flack over this outage, and rightly so. But I do want to dissuade anyone from thinking anybody else could do much better. I worked there 10 years ago, when they were closer to 200 engineers, and the caliber o…

If it's too hard for the best then the concept is dead. But I don't believe it. They made some avoidable mistakes. Just look at the pattern emerging from these kinds of incidents. There's an automatic cluster recovery mechanism that works for individual node failures but makes matters worse once a larger number of nodes fail. I wonder whether they did extensive testing or simulation of that scenario. The initial root…

The problem with all this is that you never hear about the many many failures that occur (daily, hourly at the scale they operate?) that don't cause an outage.

Re: AWS Service Disruption Post Mortem

#46

Post by kamaal »

hobbes wrote: post #6

>...one of the standard steps is to shift traffic off of one of the redundant routers in the primary EBS network to allow the upgrade to happen. The traffic shift was executed incorrectly... This supports the theory that between 50%-80% of outages are caused by human error, regardless of the resilience of the underlying infrastructure.

Unfortunately the humans in reference here are supposed to be the best of the lot. This brings in the eternal question during times of crisis ,how are the best very different than the mediocre or the average?

If not then with little hard work and smart work here and there any body can beat these 'best' during non crisis times. And during the crisis time all are same any way.

Probably that's why there are a lot of successful companies even with average talented people.

Re: AWS Service Disruption Post Mortem

#47

Post by e98cuenc »

nicpottier wrote: post #17

tldr: ""The trigger for this event was a network configuration change. We will audit our change process and increase the automation to prevent this mistake from happening in the future." AMZN has gotten a lot of flack over this outage, and rightly so. But I do want to dissuade anyone from thinking anybody else could do much better. I worked there 10 years ago, when they were closer to 200 engineers, and the caliber o…

I've been working at Google, and had a similar experience. But I do think in many cases you can actually do better. It's not that you're smarter than Amazon/Google/... engineers, it's that you're solving a problem that it's orders of magnitude easier.

Your start-up is probably not dealing with setting up & maintaining 200K servers as Amazon, so up to a limit you can actually be better going on your own.

Re: AWS Service Disruption Post Mortem

#48

Post by hobbes »

kamaal wrote: post #46
hobbes wrote: post #6

>...one of the standard steps is to shift traffic off of one of the redundant routers in the primary EBS network to allow the upgrade to happen. The traffic shift was executed incorrectly... This supports the theory that between 50%-80% of outages are caused by human error, regardless of the resilience of the underlying infrastructure.

Unfortunately the humans in reference here are supposed to be the best of the lot. This brings in the eternal question during times of crisis ,how are the best very different than the mediocre or the average? If not then with little hard work and smart work here and there any body can beat these 'best' during non crisis times. And during the crisis time all are same any way. Probably that's why there are a lot of suc…

The very best make fewer errors.

Re: AWS Service Disruption Post Mortem

#49

Post by nicpottier »

e98cuenc wrote: post #47
nicpottier wrote: post #17

tldr: ""The trigger for this event was a network configuration change. We will audit our change process and increase the automation to prevent this mistake from happening in the future." AMZN has gotten a lot of flack over this outage, and rightly so. But I do want to dissuade anyone from thinking anybody else could do much better. I worked there 10 years ago, when they were closer to 200 engineers, and the caliber o…

I've been working at Google, and had a similar experience. But I do think in many cases you can actually do better. It's not that you're smarter than Amazon/Google/... engineers, it's that you're solving a problem that it's orders of magnitude easier. Your start-up is probably not dealing with setting up & maintaining 200K servers as Amazon, so up to a limit you can actually be better going on your own.

Yep, that was my point with the 'handful of servers' line.

But even that is pretty darn hard. You still have to deal with split brain situations, do all the right things in those cases etc.. If you are a tiny startup that isn't going to hire a dedicated sysadmin (very few of which can build that type of thing without buying some expensive hardware) then EC2 is probably a better choice.

Even just the hardware costs make it pretty braindead. We just shut down the dedicated three system setup I built because it just didn't make sense financially, and that despite me spending a month learning the intricacies of heartbeat it still had odd failure scenarios that we don't experience with EC2. Again, I'm just not that smart. :)

Re: AWS Service Disruption Post Mortem

#50

Post by ekidd »

nicpottier wrote: post #17

tldr: ""The trigger for this event was a network configuration change. We will audit our change process and increase the automation to prevent this mistake from happening in the future." AMZN has gotten a lot of flack over this outage, and rightly so. But I do want to dissuade anyone from thinking anybody else could do much better. I worked there 10 years ago, when they were closer to 200 engineers, and the caliber o…

Given a choice between hosting servers on AWS, and trying to build my own reliable infrastructure with a single sysadmin, I'll take AWS in a heartbeat. But I do want to quibble with one of your points:

It is easy to be an arm chair coach and scream that... they should have predicted this storm

I'm not as smart as the AWS developers, and I have a lot less experience with large-scale distributed systems.

But thanks to my own cluelessness, I've blown up smaller distributed systems, and I've learned one important lesson: Almost nobody is smart enough to understand automatic error-recovery code. Features like automated volume remirroring or multi-AZ failover increase the load on an already stressed system, and they often cause this kind of "storm."

So I've learned to distrust intelligence in these matters. If you want to understand how your system reacts when things start going wrong, you have to find a way to simulate (or cause) large-scale failures:

This is something that Google does really really well by the way, I've watched them turn of 25 core routers simultaneously carrying hundreds of gigabits worth of data, just to verify that what they think will happen, does happen. http://news.ycombinator.com/item?id=2475112

You also need to pay particular attention to components with substantial, ongoing problems, and make sure you don't let known issues linger:

I work at Amazon EC2 and I can tell you what's going on (thanks to this handy throwaway account). What's happening is the EBS team gets inundated with support tickets due to their half-assed product. Here's the hilarious part: whenever we've asked them why they don't fix the main issue, they keep telling us that they're too busy with tickets. What they don't seem to realize is that if they fixed the core issue the tickets would go away. http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_d...

Now, I'm not saying I could have done any better than Amazon (evidence suggests otherwise). But I do know that I'm not smart enough to understand these systems without testing them to destruction, and aggressively fixing the root causes of known problems.

Post reply on HN