81–90 of 108 posts
An automatic 100% credit for 10 days usage, thats pretty good IMO
I've been noticing a trend recently when reading about large scale failures of any system: it's never just one thing. AWS EBS outage, Fukushima, Chernobyl, even the great Chicago Fire (forgive me for comparing AWS to those events). Sure there's always a "root" cause, but more importantly, it's the related events that keep adding up to make the failure even worse. I can only imagine how many minor failures happen worl…
I doubt this is the last time we'll hear of a "re-mirroring storm" in an oversaturated cloud.
Oversaturated? How do you figure? 13% were unable to re-mirror, which means 87% were able to. In short, nearly 40% of the 'cloud' was free space.
Then the 'EBS control plane' started to fail because 'slow API calls began to back up and resulted in thread starvation'. At that point, the EBS processing resources were oversaturated.
Then other nearby systems got wet.
tldr: ""The trigger for this event was a network configuration change. We will audit our change process and increase the automation to prevent this mistake from happening in the future." AMZN has gotten a lot of flack over this outage, and rightly so. But I do want to dissuade anyone from thinking anybody else could do much better. I worked there 10 years ago, when they were closer to 200 engineers, and the caliber o…
If it's too hard for the best then the concept is dead. But I don't believe it. They made some avoidable mistakes. Just look at the pattern emerging from these kinds of incidents. There's an automatic cluster recovery mechanism that works for individual node failures but makes matters worse once a larger number of nodes fail. I wonder whether they did extensive testing or simulation of that scenario. The initial root…
Earlier quoted context omitted.
Given a choice between hosting servers on AWS, and trying to build my own reliable infrastructure with a single sysadmin, I'll take AWS in a heartbeat. But I do want to quibble with one of your points: It is easy to be an arm chair coach and scream that... they should have predicted this storm I'm not as smart as the AWS developers, and I have a lot less experience with large-scale distributed systems. But thanks to…
But thanks to my own cluelessness, I've blown up smaller distributed systems, and I've learned one important lesson: Almost nobody is smart enough to understand automatic error-recovery code. Features like automated volume remirroring or multi-AZ failover increase the load on an already stressed system, and they often cause this kind of "storm." It's basically Test-Driven Development: if you cannot test it, don't wri…
Earlier quoted context omitted.
If it's too hard for the best then the concept is dead. But I don't believe it. They made some avoidable mistakes. Just look at the pattern emerging from these kinds of incidents. There's an automatic cluster recovery mechanism that works for individual node failures but makes matters worse once a larger number of nodes fail. I wonder whether they did extensive testing or simulation of that scenario. The initial root…
I think you're drawing the wrong conclusions from the grandparent post. It's not These guys are the best, so we will just have to live with these failures. It's These guys are smart people sailing in uncharted waters, so they're going to get a little lost now and then. The second conclusion implies that these waters will eventually become charted, so everyone will be able to avoid these problems in the future.
Earlier quoted context omitted.
And a breach of the "share nothing" tenet, which is quite important here.
I don't see how transparently replicated storage could be implemented without sharing something .
Did I read this correctly in paragraph 2: " For two periods during the first day of the issue, the degraded EBS cluster affected the EBS APIs and caused high error rates and latencies for EBS calls to these APIs across the entire US East Region." Their "control plane" network for the EBS clusters span availability zones in a region? If so, this would be the fatal flaw.
To me, it sounds like a large single point of failure, and the post-mortem doesn't seem to acknowledge that or discuss remedying it. They setup a separate instance of it to help with API calls in the affected region, but it still sounds like it functions across AZs and is still vulnerable overall.
"There are three things we will do to prevent a single Availability Zone from impacting the EBS control plane across multiple Availability Zones. The first is that we will immediately improve our timeout logic to prevent thread exhaustion when a single Availability Zone cluster is taking too long to process requests. … To address the cause of the second API impact, we will also add the ability for our EBS control plane to be more Availability Zone aware and shed load intelligently when it is over capacity. … Additionally, we also see an opportunity to push more of our EBS control plane into per-EBS cluster services. By moving more functionality out of the EBS control plane and creating per-EBS cluster deployments of these services (which run in the same Availability Zone as the EBS cluster they are supporting), we can provide even better Availability Zone isolation for the EBS control plane"
I've been noticing a trend recently when reading about large scale failures of any system: it's never just one thing. AWS EBS outage, Fukushima, Chernobyl, even the great Chicago Fire (forgive me for comparing AWS to those events). Sure there's always a "root" cause, but more importantly, it's the related events that keep adding up to make the failure even worse. I can only imagine how many minor failures happen worl…
