OK, it's 2019, almost 2020. Proper consumption of public cloud resources in a fashion that protects from individual Regional/AZ failures is common knowledge.
How does a post like this make it onto HN?
Amazon AWS had a power failure, their backup generators failed
21–30 of 106 posts
Re: Amazon AWS had a power failure, their backup generators failed
#22Scary that AWS can’t restore EBS volumes properly after a power failure. Snapshot are not a solution to this in a live system.
Re: Amazon AWS had a power failure, their backup generators failed
#23This seems to be getting slightly overblown in that thread. To be clear, this impacted one datacenter out of ten that make up one availability zone out of six in AWS’s us-east-1 region. So we are talking 2-3% at most of that region’s capacity was impacted. I haven’t seen a report yet on exactly why their generator failed, but from what I’ve heard, the power failed, and the backup generator kicked in and ran fine for…
We were hit by it over the weekend, and because of our failsafes, our customers didn't notice a thing. Stuff happens. It's our job to prepare for these things as well.
Re: Amazon AWS had a power failure, their backup generators failed
#24Earlier quoted context omitted.
Do you have a source detailing this issue in us-east-1? I don't see a recent post-event summary from AWS mentioning us-east-1 https://aws.amazon.com/premiumsupport/technology/pes/
Here is what they said in my support panel. This event made for a fairly tense Saturday morning for my development team and I. [01:30 PM PDT] At 4:33 AM PDT one of ten data centers in one of the six Availability Zones in the US-EAST-1 Region saw a failure of utility power. Our backup generators came online immediately but began failing at around 6:00 AM PDT. This impacted 7.5% of EC2 instances and EBS volumes in the…
Serious question, for market strategy understanding:
HA would normally use two AZs, 21C architectures would use three AZs, and other patterns such as pilot light let you use additional AZs without a significant cost hit. Further, when you can make workload-handling instances so much smaller (even down into the T sizes, and with cattle patterns you can start to leverage spot pricing), each additional AZ you add to the 21C mix represents a smaller percentage of your capacity lost in an AZ outage.
What factors have you choosing an automation powered CSP such as AWS but using only a single AZ out of a half dozen?
If using a single AZ and tense during outages, why not use Hetzner, Softlayer (now IBM Cloud), etc.?
Re: Amazon AWS had a power failure, their backup generators failed
#25[deleted]
Re: Amazon AWS had a power failure, their backup generators failed
#26So dude is mad because he didn’t have a redundancy plan? You can take snapshots of EBS volumes which backs everything up to S3. They even tell you that EBS volumes can fail in the documentation. But blaming someone else is easier I guess...
AWS need to add a button/option to ebs to have volumes be automatically backed up by aws itself. Without this few will do it or are even aware that it’s possible to do.
It doesn’t help that ebs backup takes forever, especially initially.
Re: Amazon AWS had a power failure, their backup generators failed
#27This seems to be getting slightly overblown in that thread. To be clear, this impacted one datacenter out of ten that make up one availability zone out of six in AWS’s us-east-1 region. So we are talking 2-3% at most of that region’s capacity was impacted. I haven’t seen a report yet on exactly why their generator failed, but from what I’ve heard, the power failed, and the backup generator kicked in and ran fine for…
We were hit by it over the weekend, and because of our failsafes, our customers didn't notice a thing. Stuff happens. It's our job to prepare for these things as well.
Exactly. The benefits of the cloud allow you almost unlimited computing resources spread across enormous swaths of geography, power, and internet access.
Therefore, if you were impacted, its your fault, not AWS. Sorry.
Re: Amazon AWS had a power failure, their backup generators failed
#28He is just having a temper tantrum. Everything can fail and everything does fail. There was an outage a few years ago in a well known colo/ip/managed services provider where the feeder line from the power company failed, the ATS flipped to the backup power which had a limited run time. And, due to one of those 1/1000 events that should never happen (because that ATS should flip maybe 5-6 times a year) it fused to the new position. And it happened in a place where the DC operator would cut off the service on the second line to ensure they can safely work on removing the affected ATS. So the redundant power lines + backup power did not work.
If you happened to be in that specific area of the building and happened to know building engineers and data center engineers and power company engineers you would have had heard what actually happened. Otherwise you just got "Imminent power failure" notification. Hopefully you knew that it means you want to shutdown all your workloads remotely and send someone you have on call who can reach the data center in 10-15 minutes to physically disconnect your PDUs from the incoming lines just in case someone messed up when they play at fixing the power so you don't blow 10-30% of your PDUs.
That's the reality of the life in a data center. So yeah, either accept that stuff like this happens or build for stuff like this happening. Engineering around physical problems in the cloud environment is far easier than in the data center environment.
Re: Amazon AWS had a power failure, their backup generators failed
#29Earlier quoted context omitted.
Here is what they said in my support panel. This event made for a fairly tense Saturday morning for my development team and I. [01:30 PM PDT] At 4:33 AM PDT one of ten data centers in one of the six Availability Zones in the US-EAST-1 Region saw a failure of utility power. Our backup generators came online immediately but began failing at around 6:00 AM PDT. This impacted 7.5% of EC2 instances and EBS volumes in the…
Serious question, for market strategy understanding: HA would normally use two AZs, 21C architectures would use three AZs, and other patterns such as pilot light let you use additional AZs without a significant cost hit. Further, when you can make workload-handling instances so much smaller (even down into the T sizes, and with cattle patterns you can start to leverage spot pricing), each additional AZ you add to the…
If you are using anything other than VMs, the APIs are too different.
Re: Amazon AWS had a power failure, their backup generators failed
#30So dude is mad because he didn’t have a redundancy plan? You can take snapshots of EBS volumes which backs everything up to S3. They even tell you that EBS volumes can fail in the documentation. But blaming someone else is easier I guess...
AWS need to add a button/option to ebs to have volumes be automatically backed up by aws itself. Without this few will do it or are even aware that it’s possible to do. It doesn’t help that ebs backup takes forever, especially initially.
Ah, and the storm of "my AWS bill has skyrocketed, why is AWS doing this..." incoming in 1 week after such thing. No, thanks. RTFM.