I wonder if they perform routine tests on their support infra: power, cooling, et al
No, they just let it ride and hope nothing breaks.
Amazon AWS had a power failure, their backup generators failed
11–20 of 106 posts
Re: Amazon AWS had a power failure, their backup generators failed
#12This just sounds like a bad architected solution, nothing else. The same problems that can happen in your own datacenter can happen in the cloud; it's just not your responsibility to fix it. If you lack that knowledge as an architect, rethink your title.
Re: Amazon AWS had a power failure, their backup generators failed
#13This tweet might be in response to the AWS Post event summary from August 23, 2019: > We’d like to give you some additional information about the service disruption that occurred in the Tokyo (AP-NORTHEAST-1) Region on August 23, 2019. Beginning at 12:36 PM JST, a small percentage of EC2 servers in a single Availability Zone in the Tokyo (AP-NORTHEAST-1) Region shut down due to overheating. https://aws.amazon.com/mes…
But the guy's tweet specifically mentions Reston (Virginia). This is in the vicinity of us-east-1.
By the way, I'm pretty sure none of the actual AWS datacenters are in Reston proper. They are in Ashburn and other more sparse suburbs.
Source: I live in the DC area and regularly visit Reston and Herndon. There are large AWS offices in Herndon but not so many datacenters. Real estate in Reston is pretty expensive.
Re: Amazon AWS had a power failure, their backup generators failed
#14Not only should you be architecting your app to survive an az going down you should be planning on an entire region going down and maybe even an entire cloud provider.
It’s annoying when amazon has outages but they have local outages all the time and they give you all the tools you need to handle them.
Re: Amazon AWS had a power failure, their backup generators failed
#15Is there a source for this other than an angry Twitter user?
I came across this news article: https://www.theregister.co.uk/2019/09/04/aws_power_outage_da...
The headline sounded so clickbaity that I ignored it before seeing this thread
Re: Amazon AWS had a power failure, their backup generators failed
#16This seems to be getting slightly overblown in that thread. To be clear, this impacted one datacenter out of ten that make up one availability zone out of six in AWS’s us-east-1 region. So we are talking 2-3% at most of that region’s capacity was impacted. I haven’t seen a report yet on exactly why their generator failed, but from what I’ve heard, the power failed, and the backup generator kicked in and ran fine for…
Do you have a source detailing this issue in us-east-1? I don't see a recent post-event summary from AWS mentioning us-east-1 https://aws.amazon.com/premiumsupport/technology/pes/
Here is what they said in my support panel. This event made for a fairly tense Saturday morning for my development team and I.
[01:30 PM PDT] At 4:33 AM PDT one of ten data centers in one of the six Availability Zones in the US-EAST-1 Region saw a failure of utility power. Our backup generators came online immediately but began failing at around 6:00 AM PDT. This impacted 7.5% of EC2 instances and EBS volumes in the Availability Zone. Power was fully restored to the impacted data center at 7:45 AM PDT. By 10:45 AM PDT, all but 1% of instances had been recovered, and by 12:30 PM PDT only 0.5% of instances remained impaired. Since the beginning of the impact, we have been working to recover the remaining instances and volumes. A small number of remaining instances and volumes are hosted on hardware which was adversely affected by the loss of power. We continue to work to recover all affected instances and volumes and will be communicating to the remaining impacted customers via the Personal Health Dashboard. For immediate recovery, we recommend replacing any remaining affected instances or volumes if possible.
Re: Amazon AWS had a power failure, their backup generators failed
#17Is there a source for this other than an angry Twitter user?
Here is what Amazon shared via my support panel after the outage.
[01:30 PM PDT] At 4:33 AM PDT one of ten data centers in one of the six Availability Zones in the US-EAST-1 Region saw a failure of utility power. Our backup generators came online immediately but began failing at around 6:00 AM PDT. This impacted 7.5% of EC2 instances and EBS volumes in the Availability Zone. Power was fully restored to the impacted data center at 7:45 AM PDT. By 10:45 AM PDT, all but 1% of instances had been recovered, and by 12:30 PM PDT only 0.5% of instances remained impaired. Since the beginning of the impact, we have been working to recover the remaining instances and volumes. A small number of remaining instances and volumes are hosted on hardware which was adversely affected by the loss of power. We continue to work to recover all affected instances and volumes and will be communicating to the remaining impacted customers via the Personal Health Dashboard. For immediate recovery, we recommend replacing any remaining affected instances or volumes if possible.
Re: Amazon AWS had a power failure, their backup generators failed
#18Not only should you be architecting your app to survive an az going down you should be planning on an entire region going down and maybe even an entire cloud provider. It’s annoying when amazon has outages but they have local outages all the time and they give you all the tools you need to handle them.
money money money though - all that time and effort (and fees) doesn't help you to lowball the next contract, cross your fingers and hope not to get caught out!
Re: Amazon AWS had a power failure, their backup generators failed
#19This seems to be getting slightly overblown in that thread. To be clear, this impacted one datacenter out of ten that make up one availability zone out of six in AWS’s us-east-1 region. So we are talking 2-3% at most of that region’s capacity was impacted. I haven’t seen a report yet on exactly why their generator failed, but from what I’ve heard, the power failed, and the backup generator kicked in and ran fine for…
agree.
like you imply, too many people rely heavily on their providers for business critical things like backups and redundancy. while generally big providers to a good job at this (and aws certainly does a good job at this), it does not mean there is a guarantee of any kind failures won't ever occur. thus the need to heavily invest in failure resistant technologies upon this borrowed infrastructure is arguably more important than picking 'the best' provider for the job.
I would like to note that the response time suggested by the tweet is a bit bad, 4 days to realise something and send out response / alerts to customers is a bit slow even for amazon. But then again, everything goes slower for bigger things, and amazon is quite big i'd say. not sure what the SLA response time to such an incident is, so it might be within agreed times...
Re: Amazon AWS had a power failure, their backup generators failed
#20This conclusion "The cloud is just a .. blah blah blah" is weak: Amazon offers isolated availability zones within each region to mitigate this risk at the system level, it gives you the ability to take EBS snapshots that you can backup on S3 (with 11 9s of durability), and scaling features you just will not find on "just another computer." And you are meant to architect to this with multi-AZ designs. It's managed inf…
the conclusion is weak but true. people do tend to forget the cloud is also a bunch of computers, and they might fail. however, it's not an argument to avoid cloud entirely, and in that sense its weak as an argument against cloud.