AWS Service Disruption Post Mortem
aws.amazon.com
AWS Service Disruption Post Mortem
1–10 of 108 posts
Re: AWS Service Disruption Post Mortem
#2I doubt this is the last time we'll hear of a "re-mirroring storm" in an oversaturated cloud.
Re: AWS Service Disruption Post Mortem
#3An automatic 100% credit for 10 days usage, thats pretty good IMO
Re: AWS Service Disruption Post Mortem
#4Reminds me again, Distributed System are hard and the first fallacies "The network is reliable"
Re: AWS Service Disruption Post Mortem
#5Seems there is always this issue. System fails. Systems try to repair themselves. Systems saturate something which stops them from repairing. Systems all loop aggressively bringing it all down.
Re: AWS Service Disruption Post Mortem
#6>...one of the standard steps is to shift traffic off of one of the redundant routers in the primary EBS network to allow the upgrade to happen. The traffic shift was executed incorrectly...
This supports the theory that between 50%-80% of outages are caused by human error, regardless of the resilience of the underlying infrastructure.
Re: AWS Service Disruption Post Mortem
#7I doubt this is the last time we'll hear of a "re-mirroring storm" in an oversaturated cloud.
Oversaturated? How do you figure? 13% were unable to re-mirror, which means 87% were able to. In short, nearly 40% of the 'cloud' was free space.
Re: AWS Service Disruption Post Mortem
#8An automatic 100% credit for 10 days usage, thats pretty good IMO
Well yes, except that that is usually peanuts compared to the lost income from your service being down.
Really the only purpose of a SLA penalty is to incentivize the provider to keep the network reliable.
Re: AWS Service Disruption Post Mortem
#9>...one of the standard steps is to shift traffic off of one of the redundant routers in the primary EBS network to allow the upgrade to happen. The traffic shift was executed incorrectly... This supports the theory that between 50%-80% of outages are caused by human error, regardless of the resilience of the underlying infrastructure.
Which leaves a question: why not engineer around humans, such that they are never needed in the day-to-day running of the systems?
Re: AWS Service Disruption Post Mortem
#10>...one of the standard steps is to shift traffic off of one of the redundant routers in the primary EBS network to allow the upgrade to happen. The traffic shift was executed incorrectly... This supports the theory that between 50%-80% of outages are caused by human error, regardless of the resilience of the underlying infrastructure.
Which leaves a question: why not engineer around humans, such that they are never needed in the day-to-day running of the systems?
"The trigger for this event was a network configuration change. We will audit our change process and increase the automation to prevent this mistake from happening in the future."