Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OVH outage explained

status.ovh.net

Post reply on HN
Print view View original

51–60 of 129 posts

Re: OVH outage explained

#51

Post by pbecotte »

shrike wrote: post #39

When I was at AWS we were using generators from a large commercial supplier. We were constantly having issues with them refusing to take over if there wasn't sufficient load. Doing so puts lots of stress on a generator and can significantly shorten its life. We went to the manufacturer and tried to get them to make a firmware change; we wanted the generators to sacrifice themselves under most every circumstance (shor…

Seems like you would have to disable emissions systems to get what you described, which would be illegal ;)

The actual solution to that problem, of course, is to use smaller generators?

Re: OVH outage explained

#52

Post by batrat »

duncanawoods wrote: post #4

Why are backup generators so unreliable? And that is unreliable by the standard of a non-critical system let alone that of an emergency system where reliability is their sole purpose. Is it just a (non)-survivorship bias where we most commonly talk about the failure cases and they actually have a stellar 99.99999% record that doesn't make headlines?

People. In my experience faulty generators come from bad maintenance. We have one in every site and 2 generators in locations where there is no electricity (mostly Caterpillar). They mostly fail due to lack of/bad maintenance.

Like others said, yes they are big and with lots of moving parts. But if you take care of them (every month by the book) no one should have any unexpected problems.

For me is mind blowing that OVH did the last power failure test in MAY. WHY so rare? IMO that is 1st mistake.

Re: OVH outage explained

#53

Post by patrickg_zill »

Datacenter I was using for 5 years: once per month, employees on site, power is cut/disrupted somehow, UPS kicks in, within 60 seconds the 750kw diesel genset is supposed to be running and taking load.

Zero problems in that 5 year period due to power...

Re: OVH outage explained

#54

Post by malydok »

batrat wrote: post #52
duncanawoods wrote: post #4

Why are backup generators so unreliable? And that is unreliable by the standard of a non-critical system let alone that of an emergency system where reliability is their sole purpose. Is it just a (non)-survivorship bias where we most commonly talk about the failure cases and they actually have a stellar 99.99999% record that doesn't make headlines?

People. In my experience faulty generators come from bad maintenance. We have one in every site and 2 generators in locations where there is no electricity (mostly Caterpillar). They mostly fail due to lack of/bad maintenance. Like others said, yes they are big and with lots of moving parts. But if you take care of them (every month by the book) no one should have any unexpected problems. For me is mind blowing that…

> SBG's latest test for backup recovery were at the end of May 2017. During this last test, we powered SBG only from the generators for 8 hours without any issues and every month we test the backup generators empty.

They do seem to test the generators every month.

Re: OVH outage explained

#55

Post by bluedino »

Xylakant wrote: post #6
duncanawoods wrote: post #4

Why are backup generators so unreliable? And that is unreliable by the standard of a non-critical system let alone that of an emergency system where reliability is their sole purpose. Is it just a (non)-survivorship bias where we most commonly talk about the failure cases and they actually have a stellar 99.99999% record that doesn't make headlines?

Most of the time, it's not the actual generator that fails. It's often either the part that detects power failure (fluctuations, brownouts, ... are harder to detect than total failure) or the switchover hardware to the generator. If you want to switch 20KV, you're really handling an awful lot of energy. Physics kinda get in the way there. Switching paket based networks is often much easier because you can just hold o…

>> Most of the time, it's not the actual generator that fails.

In our case, we had a generator run out of fuel. Another time we had one have the battery die (they're started by batteries just like a car engine)

Re: OVH outage explained

#56

Post by saulrh »

grecy wrote: post #49
saulrh wrote: post #46

Earlier quoted context omitted.

The generator's mechanical parts will perform more efficiently at some speeds/loads than at others. Everything happening at a different temperature and pressure will change fluid flows and forces and put different stresses in different places. Combustion won't happen cleanly, oil and gases won't flow the way they're designed to and the engine will get dirty, etc. If you're running outside of the region the engine was…

> This'd waste a ton of fuel, though, enough so that I'm not actually sure whether it'd be cheaper than eating an engine. I think you'll find fuel is extremely cheap, even when you're burning many gallons per minute.

Hmmm. I didn't actually do the numbers. Let's see...

A randomly-googled 2MW diesel generator consumes 50 gallons/hour at quarter load and 160 gallons/hour at full load [1]. So let's say it's 100 gallons/hour to run a 2 MW generator at full load instead of quarter load. The OVH incident report [OP] says that their data center has two cables in each carrying 10 MVA (mega-volts-amperes, ~ watts), giving us 20 MW as roughly the maximum power consumption of the data center. Divide, multiply, it'd cost 1000 gallons/hour to run at full load, orrrr $3000/hour at the current price of diesel [2].

Yeah, that's pretty cheap, okay. You might run these generators for a few hundred hours over the lifetime of the installation and they probably cost millions of dollars, so you're not going to spend more on fuel than you'd spend having to replace the generator early. Why don't they have a dummy load for that scenario?

[1] http://www.dieselserviceandsupply.com/Diesel_Fuel_Consumptio... [2] https://www.eia.gov/petroleum/gasdiesel/

edit: Now that I look at the math I did I realize that I didn't need to look at OVH in particular at all, just compare the cost of fuel to the purchase price of one of the diesel generators I was looking at. >_< Meh, math still works, I'll leave it.

Re: OVH outage explained

#57

Post by jotm »

exikyut wrote: post #28
AstralStorm wrote: post #26

Earlier quoted context omitted.

Apparently OVH did test their generator system but the outage detector or controller failed.

Hmm. Makes me wonder. If generators really are just a giant blob of metal and very simple (and I have no reason to disbelieve the GP comment)... well... it could be kind of interesting to build an open source software stack to handle switchover. Because, disclaimers notwithstanding, the code would ostensibly be super simple too. So even if it couldn't officially be used directly, it would certainly provide a good bas…

The actual diesel engines and generators are simple metal. Not the switching circuits. If the electric controllers can't handle the load/switching speed/whatever, no clever software and/or reliability in the actual generators will help.

Re: OVH outage explained

#58

Post by otterley »

jijji wrote: post #38

Every small datacenter has a fake diesel generator that doesn't work. I worked for one datacenter and they purposely bought a non-working diesel generator and moved it into a room with the sole purpose to pitch it to investors that they had a working secondary power source.

How on earth did you go from "I saw this thing once" to "everyone does it"?

Re: OVH outage explained

#59

Post by ngrilly »

ngrilly wrote: post #19

I have a hard time understanding why they needed humans babysitting the restart of servers and services. Servers and services are supposed to automatically restart after the power came back. This implies OVH doesn't regularly test server restarts with something like Netflix Chaos Monkey. Moreover, the power lines were not really redundant in SBG, and regarding the network downtime in RBX, it looks like the network co…

Some of you are commenting that asking OVH to randomly restart customer servers would be silly. I agree.

My comment was about servers managed by OVH. The post-mortem specifically says: "Since then, we have been working on restarting services. Powering the site with energy allows the servers to be restarted, but the services running on the servers still need to be restarted. That's why each service has been coming back gradually since 10:30 am. Our monitoring system allows us to know the list of servers that have successfully started up and those that still have a problem. We intervene on each of these servers to identify and solve the problem that prevents it from restarting."

The language used suggests that these are servers monitored and operated by OVH, not by random customers.

Re: OVH outage explained

#60

Post by gech »

grecy wrote: post #49
saulrh wrote: post #46

Earlier quoted context omitted.

The generator's mechanical parts will perform more efficiently at some speeds/loads than at others. Everything happening at a different temperature and pressure will change fluid flows and forces and put different stresses in different places. Combustion won't happen cleanly, oil and gases won't flow the way they're designed to and the engine will get dirty, etc. If you're running outside of the region the engine was…

> This'd waste a ton of fuel, though, enough so that I'm not actually sure whether it'd be cheaper than eating an engine. I think you'll find fuel is extremely cheap, even when you're burning many gallons per minute.

There's the externality of pollution you didn't consider.
Post reply on HN