Amazon AWS had a power failure, their backup generators failed

Post by darkwater » Wed, Sep 04, 2019, 12:44 PM UTC

Terretta wrote: post #24

j2bax wrote: post #16

Earlier quoted context omitted.

Here is what they said in my support panel. This event made for a fairly tense Saturday morning for my development team and I. [01:30 PM PDT] At 4:33 AM PDT one of ten data centers in one of the six Availability Zones in the US-EAST-1 Region saw a failure of utility power. Our backup generators came online immediately but began failing at around 6:00 AM PDT. This impacted 7.5% of EC2 instances and EBS volumes in the…

Serious question, for market strategy understanding: HA would normally use two AZs, 21C architectures would use three AZs, and other patterns such as pilot light let you use additional AZs without a significant cost hit. Further, when you can make workload-handling instances so much smaller (even down into the T sizes, and with cattle patterns you can start to leverage spot pricing), each additional AZ you add to the…

Post by jsjohnst » Wed, Sep 04, 2019, 12:56 PM UTC

darkwater wrote: post #31

Terretta wrote: post #24

Earlier quoted context omitted.

Serious question, for market strategy understanding: HA would normally use two AZs, 21C architectures would use three AZs, and other patterns such as pilot light let you use additional AZs without a significant cost hit. Further, when you can make workload-handling instances so much smaller (even down into the T sizes, and with cattle patterns you can start to leverage spot pricing), each additional AZ you add to the…

Can you detail/link resources about "21C architectures" and "pilot light pattern"? Thanks!

Post by acdha » Wed, Sep 04, 2019, 1:07 PM UTC

notyourday wrote: post #28

He is just having a temper tantrum. Everything can fail and everything does fail. There was an outage a few years ago in a well known colo/ip/managed services provider where the feeder line from the power company failed, the ATS flipped to the backup power which had a limited run time. And, due to one of those 1/1000 events that should never happen (because that ATS should flip maybe 5-6 times a year) it fused to the…

Post by alienreborn » Wed, Sep 04, 2019, 1:15 PM UTC

skywhopper wrote: post #7

This seems to be getting slightly overblown in that thread. To be clear, this impacted one datacenter out of ten that make up one availability zone out of six in AWS’s us-east-1 region. So we are talking 2-3% at most of that region’s capacity was impacted. I haven’t seen a report yet on exactly why their generator failed, but from what I’ve heard, the power failed, and the backup generator kicked in and ran fine for…

Post by jsjohnst » Wed, Sep 04, 2019, 1:21 PM UTC

joncrane wrote: post #13

staticvar wrote: post #9

This tweet might be in response to the AWS Post event summary from August 23, 2019: > We’d like to give you some additional information about the service disruption that occurred in the Tokyo (AP-NORTHEAST-1) Region on August 23, 2019. Beginning at 12:36 PM JST, a small percentage of EC2 servers in a single Availability Zone in the Tokyo (AP-NORTHEAST-1) Region shut down due to overheating. https://aws.amazon.com/mes…

But the guy's tweet specifically mentions Reston (Virginia). This is in the vicinity of us-east-1. By the way, I'm pretty sure none of the actual AWS datacenters are in Reston proper. They are in Ashburn and other more sparse suburbs. Source: I live in the DC area and regularly visit Reston and Herndon. There are large AWS offices in Herndon but not so many datacenters. Real estate in Reston is pretty expensive.