Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon AWS had a power failure, their backup generators failed

twitter.com

Post reply on HN
Print view View original

101–106 of 106 posts

Re: Amazon AWS had a power failure, their backup generators failed

#101

Post by jsjohnst »

lt wrote: post #42
jsjohnst wrote: post #33

Earlier quoted context omitted.

My guess on “21C architectures” is “21st century architectures”. I can’t seem to find anything on “pilot light pattern” in quick searches. Either way, GP is using obscure terminology at best.

https://d1.awsstatic.com/whitepapers/aws-disaster-recovery.p... > The term pilot light is often used to describe a DR scenario in which a minimal version of an environment is always running in the cloud. The idea of the pilot light is an analogy that comes from the gas heater. In a gas heater, a small flame that’s always on can quickly ignite the entire furnace to heat up a house

> In a gas heater, a small flame that’s always on can quickly ignite the entire furnace to heat up a house

Sure, I’m being nitpicky, but that’s a terrible explanation of the purpose of a pilot light in a legacy furnace.

Re: Amazon AWS had a power failure, their backup generators failed

#102

Post by cthalupa »

Polyisoprene wrote: post #73
Quarrel wrote: post #50

Earlier quoted context omitted.

If you lose power mid-write to an HDD, of course you can lose data. This guy sounds like if he'd self hosted he'd be complaining about an HDD failure. It happens- you need to design around it. Luckily, EBS volumes, snapshots and AZs make all of this pretty straight forward.

Data, sure. Lose the volume, no.

In my 15 years of experience as a sysadmin and architect, hard drives are far and above the most frequent hardware casualty of power failures.

The EBS documentation states that there's an expected AFR of 1 to 2 per thousand volumes, so you should plan accordingly. Replicate any data that will cause harm to your business if lost to other sources. Keep backups.

Re: Amazon AWS had a power failure, their backup generators failed

#103

Post by skywhopper »

kerkeslager wrote: post #84
skywhopper wrote: post #7

This seems to be getting slightly overblown in that thread. To be clear, this impacted one datacenter out of ten that make up one availability zone out of six in AWS’s us-east-1 region. So we are talking 2-3% at most of that region’s capacity was impacted. I haven’t seen a report yet on exactly why their generator failed, but from what I’ve heard, the power failed, and the backup generator kicked in and ran fine for…

> This seems to be getting slightly overblown in that thread. To be clear, this impacted one datacenter out of ten that make up one availability zone out of six in AWS’s us-east-1 region. So we are talking 2-3% at most of that region’s capacity was impacted. For how many users was this 100% of their business? Single-digit outage percentages for cloud services like AWS look like no big deal from the big perspective, w…

None if they had any multi-AZ capability at all.

Re: Amazon AWS had a power failure, their backup generators failed

#104

Post by inopinatus »

lovetocode wrote: post #3

So dude is mad because he didn’t have a redundancy plan? You can take snapshots of EBS volumes which backs everything up to S3. They even tell you that EBS volumes can fail in the documentation. But blaming someone else is easier I guess...

In followup, turns out they didn't even lose data as claimed.

It was a case of either lying or incompetence. Hunt didn't like that and blocked me for calling it out.

Re: Amazon AWS had a power failure, their backup generators failed

#105

Post by ti_ranger »

inopinatus wrote: post #104
lovetocode wrote: post #3

So dude is mad because he didn’t have a redundancy plan? You can take snapshots of EBS volumes which backs everything up to S3. They even tell you that EBS volumes can fail in the documentation. But blaming someone else is easier I guess...

In followup, turns out they didn't even lose data as claimed. It was a case of either lying or incompetence. Hunt didn't like that and blocked me for calling it out.

> In followup, turns out they didn't even lose data as claimed.

Link?

Re: Amazon AWS had a power failure, their backup generators failed

#106

Post by Narkov »

koolba wrote: post #34
skywhopper wrote: post #7

This seems to be getting slightly overblown in that thread. To be clear, this impacted one datacenter out of ten that make up one availability zone out of six in AWS’s us-east-1 region. So we are talking 2-3% at most of that region’s capacity was impacted. I haven’t seen a report yet on exactly why their generator failed, but from what I’ve heard, the power failed, and the backup generator kicked in and ran fine for…

EBS Snapshots don't just protect you in case of a failure to recover to something , they actually reduce the chance of failure[1]. Assuming that the S3 storage has no data loss[2], you lower the data loss possibility for your EBS volume to only those bits that have been changed since the last snapshot. [1]: https://stackoverflow.com/questions/13576363/does-taking-a-s... [2]: Yes it's a big if but S3 for durability ag…

I actually posted that Stack Overflow question and I don't think any of the answers support my assumption and your statement that it reduces the chance of failure.

The EBS volume will still fail but you can restore it from the higher reliability S3-backed EBS snapshot.

Post reply on HN