The whole thing is just too complicated to be highly-available. There will be more problems, but I wish them luck.
AWS Service Disruption Post Mortem
31–40 of 108 posts
Re: AWS Service Disruption Post Mortem
#32They should also allow one-time moves of reserved instances between availability zones.
What would the purpose of that be?
Re: AWS Service Disruption Post Mortem
#33Earlier quoted context omitted.
"The trigger for this event was a network configuration change. We will audit our change process and increase the automation to prevent this mistake from happening in the future."
Sure now, but I wanted to know why they didn't do that from the beginning.
Why they didn't do what? "Increase the automation"? I suspect that they have been doing that from the beginning. It's an ongoing process.
I hate to quote Rumsfeld, but there are known unknowns, and unknown unknowns. Of course you want to eliminate the latter-- but there's (necessarily) no way you can ever know that you've done so.
Re: AWS Service Disruption Post Mortem
#34One of the main cause for "re-mirroring storm," is node not backing off from finding a replica. Here's Twitter Back off decider implementation (Java) https://github.com/twitter/commons/blob/master/src/java/com/... When last time i looked i was little clueless on this. Now I find its usage.
Actual URL with Libya dependency removed: https://github.com/twitter/commons/blob/master/src/java/com/...
HN doesn't have a 140 character limit, so there's no need to post an obfuscated shortened link.
Re: AWS Service Disruption Post Mortem
#35tldr: ""The trigger for this event was a network configuration change. We will audit our change process and increase the automation to prevent this mistake from happening in the future." AMZN has gotten a lot of flack over this outage, and rightly so. But I do want to dissuade anyone from thinking anybody else could do much better. I worked there 10 years ago, when they were closer to 200 engineers, and the caliber o…
If it's too hard for the best then the concept is dead. But I don't believe it. They made some avoidable mistakes.
Just look at the pattern emerging from these kinds of incidents. There's an automatic cluster recovery mechanism that works for individual node failures but makes matters worse once a larger number of nodes fail.
I wonder whether they did extensive testing or simulation of that scenario. The initial root cause is probably unpredictable because there may be many, but what follows is not unpredictable.
I'm not ready to concede that because they are such an insanely smart elite group of people we just have to live with week long outages.
Re: AWS Service Disruption Post Mortem
#36Did I read this correctly in paragraph 2: " For two periods during the first day of the issue, the degraded EBS cluster affected the EBS APIs and caused high error rates and latencies for EBS calls to these APIs across the entire US East Region." Their "control plane" network for the EBS clusters span availability zones in a region? If so, this would be the fatal flaw.
And a breach of the "share nothing" tenet, which is quite important here.
Re: AWS Service Disruption Post Mortem
#37They should also allow one-time moves of reserved instances between availability zones.
What would the purpose of that be?
They want to help people make better use of multiple availability zones. People may have reserved a bunch of instances, but would be better off distributing those more effectively across zones.
Re: AWS Service Disruption Post Mortem
#38tldr: ""The trigger for this event was a network configuration change. We will audit our change process and increase the automation to prevent this mistake from happening in the future." AMZN has gotten a lot of flack over this outage, and rightly so. But I do want to dissuade anyone from thinking anybody else could do much better. I worked there 10 years ago, when they were closer to 200 engineers, and the caliber o…
If it's too hard for the best then the concept is dead. But I don't believe it. They made some avoidable mistakes. Just look at the pattern emerging from these kinds of incidents. There's an automatic cluster recovery mechanism that works for individual node failures but makes matters worse once a larger number of nodes fail. I wonder whether they did extensive testing or simulation of that scenario. The initial root…
I think it's more of a learning experience.
In the grand scheme of things, it's still day one for computing services like AWS.
Re: AWS Service Disruption Post Mortem
#39[deleted]
Re: AWS Service Disruption Post Mortem
#40"The traffic shift was executed incorrectly and rather than routing the traffic to the other router on the primary network, the traffic was routed onto the lower capacity redundant EBS network."
I'm _so_ glad I wasn't "that guy"!
(I have been "that guy" before, but never with quite such a serious fuckup multiplier in place...)