71–80 of 106 posts
Is there a source for this other than an angry Twitter user?
I came across this news article: https://www.theregister.co.uk/2019/09/04/aws_power_outage_da... The headline sounded so clickbaity that I ignored it before seeing this thread
I think the real complaint is: >>Then it took them four days to figure this out and tell us about it.
Is there some post mortem that just came out?
>>We want to give you more information on progress at this point, and what we know about the event. At 4:33 AM PDT one of 10 datacenters in one of the 6 Availability Zones in the US-EAST-1 Region saw a failure of utility power. Backup generators came online immediately, but for reasons we are still investigating, began quickly failing at around 6:00 AM PDT. This resulted in 7.5% of all instances in that Availability Zone failing by 6:10 AM PDT. Over the last few hours we have recovered most...
Scary that AWS can’t restore EBS volumes properly after a power failure. Snapshot are not a solution to this in a live system.
If you lose power mid-write to an HDD, of course you can lose data. This guy sounds like if he'd self hosted he'd be complaining about an HDD failure. It happens- you need to design around it. Luckily, EBS volumes, snapshots and AZs make all of this pretty straight forward.
This seems to be getting slightly overblown in that thread. To be clear, this impacted one datacenter out of ten that make up one availability zone out of six in AWS’s us-east-1 region. So we are talking 2-3% at most of that region’s capacity was impacted. I haven’t seen a report yet on exactly why their generator failed, but from what I’ve heard, the power failed, and the backup generator kicked in and ran fine for…
Likewise, I guess it's easy to speculate from the peanut gallery, but I wouldn't be surprised if the backup generators just hadn't been sufficiently tested and maintained because, well, they're backup generators.
Earlier quoted context omitted.
Exactly. The benefits of the cloud allow you almost unlimited computing resources spread across enormous swaths of geography, power, and internet access. Therefore, if you were impacted, its your fault, not AWS. Sorry.
* If it's within your SLA
Earlier quoted context omitted.
agree. like you imply, too many people rely heavily on their providers for business critical things like backups and redundancy. while generally big providers to a good job at this (and aws certainly does a good job at this), it does not mean there is a guarantee of any kind failures won't ever occur. thus the need to heavily invest in failure resistant technologies upon this borrowed infrastructure is arguably more…
But the reason I pay AWS is so that I don't have to hire a team to take care of backups and redundancy on my side. If they can't be relied on, a lot of the justification for their cost markup goes out the window.
This conclusion "The cloud is just a .. blah blah blah" is weak: Amazon offers isolated availability zones within each region to mitigate this risk at the system level, it gives you the ability to take EBS snapshots that you can backup on S3 (with 11 9s of durability), and scaling features you just will not find on "just another computer." And you are meant to architect to this with multi-AZ designs. It's managed inf…
the conclusion is weak but true. people do tend to forget the cloud is also a bunch of computers, and they might fail. however, it's not an argument to avoid cloud entirely, and in that sense its weak as an argument against cloud.
This seems to be getting slightly overblown in that thread. To be clear, this impacted one datacenter out of ten that make up one availability zone out of six in AWS’s us-east-1 region. So we are talking 2-3% at most of that region’s capacity was impacted. I haven’t seen a report yet on exactly why their generator failed, but from what I’ve heard, the power failed, and the backup generator kicked in and ran fine for…
AWS says they plan to install a backup fuel delivery system at this particular datacenter (it wasn't detailed whether this SPOF is common at other datacenters or if this was an outlier) and that they have already updated their notifications to be more aggressive.
Is there a reason it often seems like backup generators fail? Is it that we don't hear the success stories of all the times they don't fail? Just due to not being tested often?
It seems that it's pretty hard to get this right from the beginning too, every large datacenter ends up learning this again after they have a sequence of power incidents. That said, successful switch to generator for 1 hour and then generator failure is not a terrible outcome; if there was a notification, that's enough time to evacuate critical systems (assuming you have a plan).
We had a similar problem at our hosting facility last winter during that "Ice Vortex" storm that was all over the news. The facilities guys had been very proud of never having a power outage, I've had servers with them for 15 years now. The morning of the worst of the storm, we completely lose access to all services at that facility. Super unusual, everything we have there is redundant. So I do some minor investigati…