Amazon AWS had a power failure, their backup generators failed • Hacker News Board

Re: Amazon AWS had a power failure, their backup generators failed

#41

Post by ReidZB » Wed, Sep 04, 2019, 1:30 PM UTC

sheeshkebab wrote: post #26

lovetocode wrote: post #3

So dude is mad because he didn’t have a redundancy plan? You can take snapshots of EBS volumes which backs everything up to S3. They even tell you that EBS volumes can fail in the documentation. But blaming someone else is easier I guess...

AWS need to add a button/option to ebs to have volumes be automatically backed up by aws itself. Without this few will do it or are even aware that it’s possible to do. It doesn’t help that ebs backup takes forever, especially initially.

They did add that: https://aws.amazon.com/backup/

Re: Amazon AWS had a power failure, their backup generators failed

#42

Post by lt » Wed, Sep 04, 2019, 1:30 PM UTC

jsjohnst wrote: post #33

darkwater wrote: post #31

Earlier quoted context omitted.

Can you detail/link resources about "21C architectures" and "pilot light pattern"? Thanks!

My guess on “21C architectures” is “21st century architectures”. I can’t seem to find anything on “pilot light pattern” in quick searches. Either way, GP is using obscure terminology at best.

https://d1.awsstatic.com/whitepapers/aws-disaster-recovery.p...

> The term pilot light is often used to describe a DR scenario in which a minimal version of an environment is always running in the cloud. The idea of the pilot light is an analogy that comes from the gas heater. In a gas heater, a small flame that’s always on can quickly ignite the entire furnace to heat up a house

Re: Amazon AWS had a power failure, their backup generators failed

#43

Post by [deleted] » Wed, Sep 04, 2019, 1:35 PM UTC

darkwater wrote: post #31

Terretta wrote: post #24

Earlier quoted context omitted.

Serious question, for market strategy understanding: HA would normally use two AZs, 21C architectures would use three AZs, and other patterns such as pilot light let you use additional AZs without a significant cost hit. Further, when you can make workload-handling instances so much smaller (even down into the T sizes, and with cattle patterns you can start to leverage spot pricing), each additional AZ you add to the…

Can you detail/link resources about "21C architectures" and "pilot light pattern"? Thanks!

[deleted]

Re: Amazon AWS had a power failure, their backup generators failed

#44

Post by MichaelApproved » Wed, Sep 04, 2019, 1:43 PM UTC

nerdbaggy wrote: post #38

I remember somebody on here writing out how when they worked at AWS they wrote custom firmware for their generators to get max performance.

I couldn't find the HN post but I found an article that talks about firmware mods they do.

https://www.datacenterknowledge.com/archives/2017/04/07/how-...

> The piece of technology Amazon designed to avoid this type of outage is the firmware that decides what electrical switchgear should do when a data center loses utility power. Typical vendor firmware prioritizes preventing damage to expensive backup generators over preventing a full data center outage, according to Hamilton. Amazon (and probably most other large-scale data center operators) prefers risking the loss of a sub-$1 million piece of equipment rather than risking widespread application downtime.

> When everything happens as expected during a utility outage (which is the case most of the time), the switchgear waits a few seconds in case utility power comes back (also the most common scenario) and if it doesn’t, the switchgear fires up generators, while the data center runs on energy stored by UPS systems. Once the generators are stabilized, the switchgear makes them the primary source of power to the IT systems.

> Last year’s Delta data center outage was attributed to switchgear “locking out” the generators at the airline’s facility in Atlanta. That’s what most switchgear is designed to do when it senses a major voltage anomaly either in the data center or on the incoming utility feed. Plugging a live generator into a shorted circuit will usually fry the generator, and switchgear locks generators out to avoid that.

Re: Amazon AWS had a power failure, their backup generators failed

#45

Post by xxxpupugo » Wed, Sep 04, 2019, 1:58 PM UTC

EBS is persistent right?

I mean the failure of backup is not a surprise. Entropy is just everywhere.

Re: Amazon AWS had a power failure, their backup generators failed

#46

Post by nerdbaggy » Wed, Sep 04, 2019, 2:01 PM UTC

MichaelApproved wrote: post #44

nerdbaggy wrote: post #38

I remember somebody on here writing out how when they worked at AWS they wrote custom firmware for their generators to get max performance.

I couldn't find the HN post but I found an article that talks about firmware mods they do. https://www.datacenterknowledge.com/archives/2017/04/07/how-... > The piece of technology Amazon designed to avoid this type of outage is the firmware that decides what electrical switchgear should do when a data center loses utility power. Typical vendor firmware prioritizes preventing damage to expensive backup generators ove…

That’s it thanks!

Re: Amazon AWS had a power failure, their backup generators failed

#47

Post by therealx » Wed, Sep 04, 2019, 2:02 PM UTC

Is there a reason it often seems like backup generators fail? Is it that we don't hear the success stories of all the times they don't fail? Just due to not being tested often?

Re: Amazon AWS had a power failure, their backup generators failed

#48

Post by nwsm » Wed, Sep 04, 2019, 2:05 PM UTC

blantonl wrote: post #27

amykhar wrote: post #23

Earlier quoted context omitted.

We were hit by it over the weekend, and because of our failsafes, our customers didn't notice a thing. Stuff happens. It's our job to prepare for these things as well.

Exactly. The benefits of the cloud allow you almost unlimited computing resources spread across enormous swaths of geography, power, and internet access. Therefore, if you were impacted, its your fault, not AWS. Sorry.

* If it's within your SLA

Re: Amazon AWS had a power failure, their backup generators failed

#49

Post by skywhopper » Wed, Sep 04, 2019, 2:07 PM UTC

alienreborn wrote: post #37

skywhopper wrote: post #7

This seems to be getting slightly overblown in that thread. To be clear, this impacted one datacenter out of ten that make up one availability zone out of six in AWS’s us-east-1 region. So we are talking 2-3% at most of that region’s capacity was impacted. I haven’t seen a report yet on exactly why their generator failed, but from what I’ve heard, the power failed, and the backup generator kicked in and ran fine for…

Some AWS services can't span across multiple availability zones like EMR.

Yes, and also for many managed services (like Kinesis or Lambda) you don't have any control over which AZs are in use, and so you have no choice but to wait for the AWS teams to address any problems caused on their own backends.

Regardless, it's important to be aware of the risks of whatever tool you're using. It's unrealistic to expect any provider to be able to avoid failure entirely. You have to be aware of possible failure scenarios and have your own plan to address them.

Re: Amazon AWS had a power failure, their backup generators failed

#50

Post by Quarrel » Wed, Sep 04, 2019, 2:14 PM UTC

Polyisoprene wrote: post #22

Scary that AWS can’t restore EBS volumes properly after a power failure. Snapshot are not a solution to this in a live system.

If you lose power mid-write to an HDD, of course you can lose data.

This guy sounds like if he'd self hosted he'd be complaining about an HDD failure. It happens- you need to design around it. Luckily, EBS volumes, snapshots and AZs make all of this pretty straight forward.