Amazon AWS had a power failure, their backup generators failed
1–10 of 106 posts
Re: Amazon AWS had a power failure, their backup generators failed
#2I wonder if they perform routine tests on their support infra: power, cooling, et al
Re: Amazon AWS had a power failure, their backup generators failed
#3So dude is mad because he didn’t have a redundancy plan? You can take snapshots of EBS volumes which backs everything up to S3. They even tell you that EBS volumes can fail in the documentation. But blaming someone else is easier I guess...
Re: Amazon AWS had a power failure, their backup generators failed
#4Is there a source for this other than an angry Twitter user?
Re: Amazon AWS had a power failure, their backup generators failed
#5I wonder if they perform routine tests on their support infra: power, cooling, et al
Yes and frequently.
Re: Amazon AWS had a power failure, their backup generators failed
#6I wonder if they perform routine tests on their support infra: power, cooling, et al
Wonder no more, the answer is: yes, obviously
Re: Amazon AWS had a power failure, their backup generators failed
#7This seems to be getting slightly overblown in that thread. To be clear, this impacted one datacenter out of ten that make up one availability zone out of six in AWS’s us-east-1 region. So we are talking 2-3% at most of that region’s capacity was impacted.
I haven’t seen a report yet on exactly why their generator failed, but from what I’ve heard, the power failed, and the backup generator kicked in and ran fine for over an hour, but then it failed. This sort of thing sucks to deal with but it’s also inevitable. Of their hundreds of datacenters, a mechanical failure is going to happen occasionally no matter how good their maintenance plans are.
So the key when using cloud services like AWS is to plan for the possibility of failure. EBS expects an annual failure rate of 0.1%. So one out of a thousand EBS volumes will fail in a given year. If you operate at the scale of thousands of servers in AWS, you see this sort of thing all the time. Luckily, EBS also makes it trivial to take volume snapshots which are stored in S3, which has much much higher reliability and durability. So if you have data in EBS that needs to be kept safe, take regular snapshots. Here’s a doc that explains how you can set up scheduled, auto-rotated snapshots: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/snapshot...
Re: Amazon AWS had a power failure, their backup generators failed
#8This conclusion "The cloud is just a .. blah blah blah" is weak: Amazon offers isolated availability zones within each region to mitigate this risk at the system level, it gives you the ability to take EBS snapshots that you can backup on S3 (with 11 9s of durability), and scaling features you just will not find on "just another computer." And you are meant to architect to this with multi-AZ designs.
It's managed infrastructure not some miraculous alternative universe where probabilities do not apply to you.
From the docs:
"Amazon EBS volumes are designed for an annual failure rate (AFR) of between 0.1% - 0.2% ..."
Re: Amazon AWS had a power failure, their backup generators failed
#9This tweet might be in response to the AWS Post event summary from August 23, 2019:
> We’d like to give you some additional information about the service disruption that occurred in the Tokyo (AP-NORTHEAST-1) Region on August 23, 2019. Beginning at 12:36 PM JST, a small percentage of EC2 servers in a single Availability Zone in the Tokyo (AP-NORTHEAST-1) Region shut down due to overheating.
Re: Amazon AWS had a power failure, their backup generators failed
#10This seems to be getting slightly overblown in that thread. To be clear, this impacted one datacenter out of ten that make up one availability zone out of six in AWS’s us-east-1 region. So we are talking 2-3% at most of that region’s capacity was impacted. I haven’t seen a report yet on exactly why their generator failed, but from what I’ve heard, the power failed, and the backup generator kicked in and ran fine for…
Do you have a source detailing this issue in us-east-1? I don't see a recent post-event summary from AWS mentioning us-east-1 https://aws.amazon.com/premiumsupport/technology/pes/