Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OVH outage explained

status.ovh.net

Post reply on HN
Print view View original

91–100 of 129 posts

Re: OVH outage explained

#91

Post by staunch »

duncanawoods wrote: post #4

Why are backup generators so unreliable? And that is unreliable by the standard of a non-critical system let alone that of an emergency system where reliability is their sole purpose. Is it just a (non)-survivorship bias where we most commonly talk about the failure cases and they actually have a stellar 99.99999% record that doesn't make headlines?

Because it's "backup" hardware. Like a lot of insurance, it's often just a false sense of security.

It would be interesting to run a datacenter completely on batteries that are charged from solar + the grid + gas generators. You could run the generators at optimal power output and use them daily. The batteries required for ~24-48 hours might still be too expensive to make this possible. Maybe some kind of super low power datacenter could pull this off. One day it shall be mine.

Re: OVH outage explained

#92

Post by ChuckMcM »

jcrawfordor wrote: post #80
etaioinshrdlu wrote: post #68

Earlier quoted context omitted.

It seems like excessively low load would be a relatively easy problem to solve. One could heat tanks of water to boiling and vent the steam, that could probably absorb as much energy as needed. Or just crank up the AC :)

There are load banks made for generators, just large resistive grids. A facility that I work at has one so that we can exercise the generator at operating load without actually switching over our power. So I would think a good solution for a DC might be to have load banks you could switch in when necessary.

Somewhere I have a picture of a car that was parked in the no parking area next to the dummy load for a data center generator. All of the plastic parts on the outside are melted off and the paint is a different color on the side that got the bulk of the hot air coming off the dummy load.

I had not realized it could get pretty hot there but it made sense given the energy they were dumping out.

Re: OVH outage explained

#93

Post by cesarb »

Cyph0n wrote: post #36
kyledrake wrote: post #30

My datacenter has 2 power channels, coming from two different power grids. They have never had a full outage, even when a substation two blocks away started on fire. Even if we lost power on one power channel, we would still be operational. Even if the backup power failed. That's the reason you use redundant power. If your "cloud provider" is charging you a lot but not providing your server redundant power from two g…

The power grid in and of itself is a highly interconnected and redundant system. The fact that a substation fire didn't bring down your datacenter doesn't necessarily mean that the "redundant" setup was responsible. Unless the two grids are fully isolated (e.g., from two different countries), there is always going to be some overlap in the two power sources.

Even two different countries is no guarantee of isolation. The 2009 blackout affected both Paraguay and a large portion of Brazil. If two countries are near enough that it's feasible to get power from both, they're near enough that their power networks might be interconnected.

Re: OVH outage explained

#94

Post by cesarb »

ngrilly wrote: post #78
snom380 wrote: post #73

Earlier quoted context omitted.

I agree it's a bug, but this is far, far more difficult to test against with a chaos monkey than one might think. With a whole datacenter down, not to speak of multiple availability zones, problems and dependencies will crop up that are extremely difficult to anticipate. Maybe you expected one other AZ to be up so you could provision servers from there, but now you have to somehow get one of them bootstrapped. Or may…

Thanks for the great explanation. I agree that a Chaos Monkey test something a lot more easier to recover from than a whole datacenter getting down. I remember having read somewhere about some company (Facebook/Google/Amazon/Twitter/Dropbox or something at the same scale) that regularly simulates a whole datacenter failure, which made believe it is possible to automatically recover from this. Are you saying that even…

> I remember having read somewhere about some company (Facebook/Google/Amazon/Twitter/Dropbox or something at the same scale) that regularly simulates a whole datacenter failure

It was Google: http://queue.acm.org/detail.cfm?id=2371516 ("Weathering the Unexpected" [2012])

Re: OVH outage explained

#95

Post by DannyDaemonic »

jankey wrote: post #72
etaioinshrdlu wrote: post #68

Earlier quoted context omitted.

It seems like excessively low load would be a relatively easy problem to solve. One could heat tanks of water to boiling and vent the steam, that could probably absorb as much energy as needed. Or just crank up the AC :)

Huge DCs like Amazons and ours at Google are very efficient, cooling uses at most 10% of total power. So cranking up this wouldn't help. Nor wouldn't it be possible usual anyway, as these are typically using evaporation cooling, which can't really be cranked up like traditional compression cooling. And keeping a huge tank and heater ready would introduce another point of failure. The easiest is just to have your serv…

Compared to heating a water tank, it would be a much more responsible use of power to fire up some folding@home images.

Perhaps a more financially responsible solution would be to spin up a bunch of instances that mine some sort of cryptocurrency. I doubt it'd cover electricity costs, but it could offset it some.

Re: OVH outage explained

#96

Post by dboreham »

ngrilly wrote: post #19

I have a hard time understanding why they needed humans babysitting the restart of servers and services. Servers and services are supposed to automatically restart after the power came back. This implies OVH doesn't regularly test server restarts with something like Netflix Chaos Monkey. Moreover, the power lines were not really redundant in SBG, and regarding the network downtime in RBX, it looks like the network co…

Meanwhile in the real world...

Re: OVH outage explained

#97

Post by mrep »

kittens321 wrote: post #76
shrike wrote: post #39

When I was at AWS we were using generators from a large commercial supplier. We were constantly having issues with them refusing to take over if there wasn't sufficient load. Doing so puts lots of stress on a generator and can significantly shorten its life. We went to the manufacturer and tried to get them to make a firmware change; we wanted the generators to sacrifice themselves under most every circumstance (shor…

I'm not an expert, but a high power device safe load means that it won't place a lot of the load on itself. when it places the load on itself it gets really hot, so hot that parts may melt, and things that were supposed to be insulated may now be conductive. Doing high voltage electronics without a license is a crime in a lot of places. I'm pretty sure a person with the right specialty can add a load so you won't nee…

Amazon spends billions on energy and has hundreds of hardware and electrical engineers. They are one of the best if not the best in data center engineering. They know what they are doing.

Re: OVH outage explained

#98

Post by js2 »

shrike wrote: post #39

When I was at AWS we were using generators from a large commercial supplier. We were constantly having issues with them refusing to take over if there wasn't sufficient load. Doing so puts lots of stress on a generator and can significantly shorten its life. We went to the manufacturer and tried to get them to make a firmware change; we wanted the generators to sacrifice themselves under most every circumstance (shor…

> we wanted the generators to sacrifice themselves under most every circumstance

The rare failure mode that can cost $100m looks like this: when the utility power fails, the switch gear detects a voltage anomaly sufficiently large to indicate a high probability of a ground fault within the data center. A generator brought online into a direct short could be damaged. With expensive equipment possibly at risk, the switch gear locks out the generator. Five to ten minutes after that decision, the UPS will discharge and row after row of servers will start blinking out. This same fault mode caused the 34-minute outage at the 2012 super bowl: The Power Failure Seen Around the World. Backup generators run around 3/4 of million dollars so I understand the switch gear engineering decision to lockout and protect an expensive component. And, while I suspect that some customers would want it that way, I’ve never worked for one of those customers and the airline hit by this fault last summer certainly isn’t one of them either.

http://perspectives.mvdirona.com/2017/04/at-scale-rare-event...

Re: OVH outage explained

#99

Post by darklajid »

scurvy wrote: post #81
sphix0r wrote: post #7

Earlier quoted context omitted.

Similar to many of us make backups in IT but how many actually do try them out and restore?

waves hand Every day. All of them (automated).

Can you turn/expand that into a somewhat longer answer, a list of links that you perused to build that or .. a post on its own?

Re: OVH outage explained

#100

Post by rasz »

js2 wrote: post #98
shrike wrote: post #39

When I was at AWS we were using generators from a large commercial supplier. We were constantly having issues with them refusing to take over if there wasn't sufficient load. Doing so puts lots of stress on a generator and can significantly shorten its life. We went to the manufacturer and tried to get them to make a firmware change; we wanted the generators to sacrifice themselves under most every circumstance (shor…

> we wanted the generators to sacrifice themselves under most every circumstance The rare failure mode that can cost $100m looks like this: when the utility power fails, the switch gear detects a voltage anomaly sufficiently large to indicate a high probability of a ground fault within the data center. A generator brought online into a direct short could be damaged. With expensive equipment possibly at risk, the swit…

This sounds nice, but makes zero sense. Direct short will be fused and isolated, otherwise UPS banks would get damaged (maybe even explode).

Edit, and reading further there it is:

>"If there was a ground fault in the facility, the impacted branch circuit breaker would open and the rest of the facility would continue to operate on generator and the servers downstream of the open breaker would switch to secondary power and also continue to operate normally. No customer impact."

Post reply on HN