Live data from Hacker News

Amazon AWS had a power failure, their backup generators failed

twitter.com

61–70 of 106 posts

Re: Amazon AWS had a power failure, their backup generators failed

#61
post #23

Earlier quoted context omitted.

We were hit by it over the weekend, and because of our failsafes, our customers didn't notice a thing. Stuff happens. It's our job to prepare for these things as well.

Curious to know what kind of failsafes you have in place so your customers were not impacted?

probably multi-az, multi-tier, stateless apps

https://12factor.net/ can help guide

Re: Amazon AWS had a power failure, their backup generators failed

#62
post #23

Earlier quoted context omitted.

We were hit by it over the weekend, and because of our failsafes, our customers didn't notice a thing. Stuff happens. It's our job to prepare for these things as well.

Curious to know what kind of failsafes you have in place so your customers were not impacted?

I'm not the original commenter but for an EC2 hosted application, any architecture that uses a VPC with proper cross-AZ subnetting and a load balancer would not have been impacted. EBS snapshots would minimize any data loss.

Re: Amazon AWS had a power failure, their backup generators failed

#63

Earlier quoted context omitted.

agree. like you imply, too many people rely heavily on their providers for business critical things like backups and redundancy. while generally big providers to a good job at this (and aws certainly does a good job at this), it does not mean there is a guarantee of any kind failures won't ever occur. thus the need to heavily invest in failure resistant technologies upon this borrowed infrastructure is arguably more…

But the reason I pay AWS is so that I don't have to hire a team to take care of backups and redundancy on my side. If they can't be relied on, a lot of the justification for their cost markup goes out the window.

It can be relied on, but it's up to you to configure it properly. AWS has no way of knowing how critical your application is and what level of redundancy it needs, and this has cost implications so they can't do it automatically.

Re: Amazon AWS had a power failure, their backup generators failed

#64

Earlier quoted context omitted.

agree. like you imply, too many people rely heavily on their providers for business critical things like backups and redundancy. while generally big providers to a good job at this (and aws certainly does a good job at this), it does not mean there is a guarantee of any kind failures won't ever occur. thus the need to heavily invest in failure resistant technologies upon this borrowed infrastructure is arguably more…

But the reason I pay AWS is so that I don't have to hire a team to take care of backups and redundancy on my side. If they can't be relied on, a lot of the justification for their cost markup goes out the window.

If you don't want to think about things like redundancy then use higher-abstraction services. Lambda for example takes care of multi-AZ redundancy so you don't have to think about it. The lower level building blocks like EC2 don't. They expose the fault boundaries so that you can build HA applications on top of them, but it's still your responsibility to do so.

Re: Amazon AWS had a power failure, their backup generators failed

#65

Earlier quoted context omitted.

But the reason I pay AWS is so that I don't have to hire a team to take care of backups and redundancy on my side. If they can't be relied on, a lot of the justification for their cost markup goes out the window.

It can be relied on, but it's up to you to configure it properly. AWS has no way of knowing how critical your application is and what level of redundancy it needs, and this has cost implications so they can't do it automatically.

I'm pretty sure that RDS is EBS-backed.

Re: Amazon AWS had a power failure, their backup generators failed

#66
Is this North Virginia AGAIN???? How ironic that a company named "Amazon" cannot keep it's servers up when ever there is rain. This has happened practically every time hurricane season appears. Obviously being a bit hard on them (for humor) but come on guys, get a giant umbrella or something.

Re: Amazon AWS had a power failure, their backup generators failed

#67

I remember somebody on here writing out how when they worked at AWS they wrote custom firmware for their generators to get max performance.

I couldn't find the HN post but I found an article that talks about firmware mods they do. https://www.datacenterknowledge.com/archives/2017/04/07/how-... > The piece of technology Amazon designed to avoid this type of outage is the firmware that decides what electrical switchgear should do when a data center loses utility power. Typical vendor firmware prioritizes preventing damage to expensive backup generators ove…

Diesel generators at Hospitals and diesel motors running pumps for fire suppression systems are normally set up to keep running closer to the line of risking damage to the generator and engine.

Re: Amazon AWS had a power failure, their backup generators failed

#68

Earlier quoted context omitted.

It can be relied on, but it's up to you to configure it properly. AWS has no way of knowing how critical your application is and what level of redundancy it needs, and this has cost implications so they can't do it automatically.

I'm pretty sure that RDS is EBS-backed.

… and RDS has a multi-AZ checkbox which does exactly what it claims. Anyone who used it did not have a problem with this outage.

Re: Amazon AWS had a power failure, their backup generators failed

#69

This seems to be getting slightly overblown in that thread. To be clear, this impacted one datacenter out of ten that make up one availability zone out of six in AWS’s us-east-1 region. So we are talking 2-3% at most of that region’s capacity was impacted. I haven’t seen a report yet on exactly why their generator failed, but from what I’ve heard, the power failed, and the backup generator kicked in and ran fine for…

Some AWS services can't span across multiple availability zones like EMR.

EMR seems kind of legacy from AWS's perspective these days. Probably very profitable, but they're seeming to try to push you towards things like Glue and Athena.

Re: Amazon AWS had a power failure, their backup generators failed

#70

Earlier quoted context omitted.

agree. like you imply, too many people rely heavily on their providers for business critical things like backups and redundancy. while generally big providers to a good job at this (and aws certainly does a good job at this), it does not mean there is a guarantee of any kind failures won't ever occur. thus the need to heavily invest in failure resistant technologies upon this borrowed infrastructure is arguably more…

But the reason I pay AWS is so that I don't have to hire a team to take care of backups and redundancy on my side. If they can't be relied on, a lot of the justification for their cost markup goes out the window.

Respectfully, that's not a good reason to use public cloud providers like AWS. They provide features and tooling that make building redundancy into your services easier but for many of these redundancy features you must integrate them into your infrastructure design to take advantage of them.
Post reply on HN