Compare to the 2008 post-mortem: http://status.aws.amazon.com/s3-20080720.html Messaging infrastructure as single point of failure? Check. http://news.ycombinator.com/item?id=2472227
AWS Service Disruption Post Mortem
61–70 of 108 posts
Re: AWS Service Disruption Post Mortem
#62tldr: ""The trigger for this event was a network configuration change. We will audit our change process and increase the automation to prevent this mistake from happening in the future." AMZN has gotten a lot of flack over this outage, and rightly so. But I do want to dissuade anyone from thinking anybody else could do much better. I worked there 10 years ago, when they were closer to 200 engineers, and the caliber o…
See, this makes me uneasy. The best and brightest work very hard to avoid memory errors in their C codebases then, when errors occur anyway, console themselves that no-one could have done any better. The less brilliant look into more automatic forms of memory management. AT&T hired the best and brightest to make heroic efforts so that the old circuit-switched Bell system could achieve mediocre reliability. I certainly hope that the best and brightest at AWS aren't intending to tackle their enormous control-backplane SPOF by assiduously patching every bug that turns up in its behaviour.
Re: AWS Service Disruption Post Mortem
#63Did I read this correctly in paragraph 2: " For two periods during the first day of the issue, the degraded EBS cluster affected the EBS APIs and caused high error rates and latencies for EBS calls to these APIs across the entire US East Region." Their "control plane" network for the EBS clusters span availability zones in a region? If so, this would be the fatal flaw.
And a breach of the "share nothing" tenet, which is quite important here.
I don't see how transparently replicated storage could be implemented without sharing something.
Re: AWS Service Disruption Post Mortem
#64Did I read this correctly in paragraph 2: " For two periods during the first day of the issue, the degraded EBS cluster affected the EBS APIs and caused high error rates and latencies for EBS calls to these APIs across the entire US East Region." Their "control plane" network for the EBS clusters span availability zones in a region? If so, this would be the fatal flaw.
Isn't this "flaw" a balance of features vs reliability? In most cases it's ok that the API spans the entire region, and makes it easy to address one API endpoint per region, allows Amazon to offer everyone different availibility zones within the region, etc.
The issue being that availability zones don't get throttled when they start to overload the API. The control system should have automatically throttled the misbehaving AV.
Which is probably far more difficult to do properly than I can imagine.
Re: AWS Service Disruption Post Mortem
#65Earlier quoted context omitted.
And a breach of the "share nothing" tenet, which is quite important here.
I don't see how transparently replicated storage could be implemented without sharing something .
In which case they shouldn't replicate across availability zones.
'shared nothing' is the only way to islandize failures.
Re: AWS Service Disruption Post Mortem
#66The whole thing is just too complicated to be highly-available. There will be more problems, but I wish them luck.
That's a bit defeatist. In my first year of uni, one of our lecturers drummed something into us:
Q. How do you eat an elephant?
A. One bite at a time
Re: AWS Service Disruption Post Mortem
#67I've been noticing a trend recently when reading about large scale failures of any system: it's never just one thing.
AWS EBS outage, Fukushima, Chernobyl, even the great Chicago Fire (forgive me for comparing AWS to those events).
Sure there's always a "root" cause, but more importantly, it's the related events that keep adding up to make the failure even worse. I can only imagine how many minor failures happen world wide on a daily basis where there's only a root cause and no further chain of events.
Once a system is sufficiently complex, I'm not sure it's possible to make it completely fault-tolerant. I'm starting to believe that there's always some chain of events which would lead to a massive failure. And the more complex a system is, the more "chains of failure" exist. It would also become increasingly difficult to plan around failures.
edit: The Logic of Failure is recommended to anyone wanted to know more about this subject: http://www.amazon.com/Logic-Failure-Recognizing-Avoiding-Sit...
Re: AWS Service Disruption Post Mortem
#68"We will look to provide customers with better tools to create multi-AZ applications that can support the loss of an entire Availability Zone without impacting application availability. We know we need to help customers design their application logic using common design patterns. In this event, some customers were seriously impacted, and yet others had resources that were impacted but saw nearly no impact on their applications."
Re: AWS Service Disruption Post Mortem
#69This article reads like nonsense - but this is not a criticism of AWS. The real problem is there is no good mathematical model of distributed behaviour, from which statistical guarantees can be made. I think we're at the limit of what the smartest people can achieve with hand crafted code. Most likely new math will give rise to new tools and languages, in which the next generation of reliable distributed systems will…
I'm curious - what about this post-mortem "reads like nonsense"?
Re: AWS Service Disruption Post Mortem
#70There's a quote I found interesting that hasn't been noted here yet:
"This required the time-consuming process of physically relocating excess server capacity from across the US East Region and installing that capacity into the degraded EBS cluster."
And if I read this description of the re-mirror storm correctly, I think that implies Amazon had to increase the size of it's EBS cluster in the affected zone by 13%, which considering the timeline seems fairly impressive.