mLab was affected by this, and based on their own status page it looks like some volumes are permanently unrecoverable: https://status.mlab.com/
Amazon AWS had a power failure, their backup generators failed
51–60 of 106 posts
Re: Amazon AWS had a power failure, their backup generators failed
#52Is there a reason it often seems like backup generators fail? Is it that we don't hear the success stories of all the times they don't fail? Just due to not being tested often?
If you were buying datacenter space instead of cloud resources, you would expect each datacenter to report to you quarterly that they had run their backup generator tests and fixed any problems that occurred. You would also expect to be told in advance about network maintenance and other foreseeable service issues.
Cloud vendors, of course, get to hide all that from you until something goes wrong.
Re: Amazon AWS had a power failure, their backup generators failed
#53This conclusion "The cloud is just a .. blah blah blah" is weak: Amazon offers isolated availability zones within each region to mitigate this risk at the system level, it gives you the ability to take EBS snapshots that you can backup on S3 (with 11 9s of durability), and scaling features you just will not find on "just another computer." And you are meant to architect to this with multi-AZ designs. It's managed inf…
the conclusion is weak but true. people do tend to forget the cloud is also a bunch of computers, and they might fail. however, it's not an argument to avoid cloud entirely, and in that sense its weak as an argument against cloud.
exactly. remember, everything fails, all the time. https://thenextweb.com/2008/04/04/werner-vogels-everything-f...
Re: Amazon AWS had a power failure, their backup generators failed
#54We had a similar problem at our hosting facility last winter during that "Ice Vortex" storm that was all over the news. The facilities guys had been very proud of never having a power outage, I've had servers with them for 15 years now.
The morning of the worst of the storm, we completely lose access to all services at that facility. Super unusual, everything we have there is redundant. So I do some minor investigation, and get on the horn to them.
They were being super cagey. "Hey, we lost all access to our systems." "Ok, I'll open a ticket and we will investigate." "Uhhh. It feels like it's a big problem with the data center, are you guys having problems or is it just us?" "I can't say anything more until we've completed an investigation." "I'm trying to decide if we need to start failing over to our DR site, or if I need to put chains on the truck to drive 50 miles to the data center in this storm. Is anyone else having problems? Are fire alarms going off?" "We have received multiple reports of problems."
Power was back on in less than half an hour, but they still weren't saying anything for a few hours. Spent that time trying to figure out if we should shut everything down and ride it out, or if they were back in business. We had one system that suffered disk corruption, despite having a (according to the weekly testing) correctly operating BBU on the RAID.
So what happened? It shouldn't have been possible, our cabinet was being fed by two lines from independent PDUs. Each PDU is fed by 2 independent UPSes (one shared between the two PDUs), each UPS fed by a dedicated generator. Should have required 3 failures to bring us down.
They eventually reveal that they had had one of their UPSes down for weeks, waiting for replacement parts. The other two UPSes had independent failures (one was a controller board, one was battery related). They said they still did quarterly full load tests of the power systems, but reading between the lines I think they weren't testing these two UPSes because the other one was not there to back them up.
Still, one power event in 15 years isn't too shabby.
Re: Amazon AWS had a power failure, their backup generators failed
#55This seems to be getting slightly overblown in that thread. To be clear, this impacted one datacenter out of ten that make up one availability zone out of six in AWS’s us-east-1 region. So we are talking 2-3% at most of that region’s capacity was impacted. I haven’t seen a report yet on exactly why their generator failed, but from what I’ve heard, the power failed, and the backup generator kicked in and ran fine for…
agree. like you imply, too many people rely heavily on their providers for business critical things like backups and redundancy. while generally big providers to a good job at this (and aws certainly does a good job at this), it does not mean there is a guarantee of any kind failures won't ever occur. thus the need to heavily invest in failure resistant technologies upon this borrowed infrastructure is arguably more…
But the reason I pay AWS is so that I don't have to hire a team to take care of backups and redundancy on my side. If they can't be relied on, a lot of the justification for their cost markup goes out the window.
Re: Amazon AWS had a power failure, their backup generators failed
#56I wonder if they perform routine tests on their support infra: power, cooling, et al
No, they just let it ride and hope nothing breaks.
I know you're being sarcastic but you're probably more right than you realize. Many many "redundant" systems turn out to not be so redundant when it counts.
talk to some datacenter admins and you'll learn there's a lot more bailing wire and hope-for-the-best out there than you would think.
Re: Amazon AWS had a power failure, their backup generators failed
#57mLab was affected by this, and based on their own status page it looks like some volumes are permanently unrecoverable: https://status.mlab.com/
I can confirm that. We had a single-AZ RDS instance whose underlying storage (a "magnetic" EBS volume) unrecoverably failed, according to AWS. It had to be restored from a backup. Fortunately, "point-in-time recovery" meant there was very little data loss, just some downtime.
(Not that data loss or downtime mattered for this instance, which was just used for internal testing.)
Power-up is very stressful for hard drives, so it's not too surprising that some failed when the power turned back on. EBS does offer spinning rust storage options, so maybe mLab was using those for some of those failed volumes. I don't know if the same is true for SSDs or not.
Re: Amazon AWS had a power failure, their backup generators failed
#58This seems to be getting slightly overblown in that thread. To be clear, this impacted one datacenter out of ten that make up one availability zone out of six in AWS’s us-east-1 region. So we are talking 2-3% at most of that region’s capacity was impacted. I haven’t seen a report yet on exactly why their generator failed, but from what I’ve heard, the power failed, and the backup generator kicked in and ran fine for…
We were hit by it over the weekend, and because of our failsafes, our customers didn't notice a thing. Stuff happens. It's our job to prepare for these things as well.
Curious to know what kind of failsafes you have in place so your customers were not impacted?
Re: Amazon AWS had a power failure, their backup generators failed
#59Is there a reason it often seems like backup generators fail? Is it that we don't hear the success stories of all the times they don't fail? Just due to not being tested often?
Yes, you don’t hear the hundred or two times they start correctly, as they typically are tested quarterly to monthly. A good test is where you island the datacenter (run it only off the generators) so you can also make sure all the switchgear works.
Re: Amazon AWS had a power failure, their backup generators failed
#60I think the real complaint is: >>Then it took them four days to figure this out and tell us about it.
I mean what did he expect? A 5 min resolution? 4 days sounds like a very reasonable investigation time to figure out why something complex failed.