Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OVH outage explained

status.ovh.net

Post reply on HN
Print view View original

61–70 of 129 posts

Re: OVH outage explained

#61

Post by ngrilly »

vbernat wrote: post #35
ngrilly wrote: post #19

I have a hard time understanding why they needed humans babysitting the restart of servers and services. Servers and services are supposed to automatically restart after the power came back. This implies OVH doesn't regularly test server restarts with something like Netflix Chaos Monkey. Moreover, the power lines were not really redundant in SBG, and regarding the network downtime in RBX, it looks like the network co…

Restarting a whole datacenter from scratch is different from restarting individual servers. If a service needs another service running on another server, it may fail to start. For example, many servers may need the DHCP server. Some servers may need the DNS. Some others may need Kafka to be up and running, etc. You may have some documentation on which servers to start first, but things are moving so fast that it's no…

I don't buy this argument. If a service needs another service running on another server which is not already available, it should retry with an exponential backoff. Doing anything else is a bug.

Re: OVH outage explained

#62

Post by danmaz74 »

gech wrote: post #60
grecy wrote: post #49

Earlier quoted context omitted.

> This'd waste a ton of fuel, though, enough so that I'm not actually sure whether it'd be cheaper than eating an engine. I think you'll find fuel is extremely cheap, even when you're burning many gallons per minute.

There's the externality of pollution you didn't consider.

That's true, but how much pollution does it cost to replace the generator?

Re: OVH outage explained

#63

Post by rdl »

saulrh wrote: post #56
grecy wrote: post #49

Earlier quoted context omitted.

> This'd waste a ton of fuel, though, enough so that I'm not actually sure whether it'd be cheaper than eating an engine. I think you'll find fuel is extremely cheap, even when you're burning many gallons per minute.

Hmmm. I didn't actually do the numbers. Let's see... A randomly-googled 2MW diesel generator consumes 50 gallons/hour at quarter load and 160 gallons/hour at full load [1]. So let's say it's 100 gallons/hour to run a 2 MW generator at full load instead of quarter load. The OVH incident report [OP] says that their data center has two cables in each carrying 10 MVA (mega-volts-amperes, ~ watts), giving us 20 MW as roug…

Also, if you have one power outage, it's more likely you'll have another soon, so you don't want to burn up your generator (likely a long lead time to replace) during the first outage, just to have nothing during the next outage.

Re: OVH outage explained

#64

Post by madez »

shrike wrote: post #39

When I was at AWS we were using generators from a large commercial supplier. We were constantly having issues with them refusing to take over if there wasn't sufficient load. Doing so puts lots of stress on a generator and can significantly shorten its life. We went to the manufacturer and tried to get them to make a firmware change; we wanted the generators to sacrifice themselves under most every circumstance (shor…

> AWS really likes to own the entire stack.

I wish you and I could own the full stack on our hardware, too. Imagine free drivers, firmware and microcode, full schematics and specification of all the parts. That would approach my atheist's heaven.

Re: OVH outage explained

#65

Post by cperciva »

gedrap wrote: post #37
cesarb wrote: post #34

I wonder how easy it is to really test these failover systems. Even if you disconnect the external power (which is risky, since you lose redundancy), it will be a "clean" disconnection, with the power going neatly to zero. I've found a few post-mortems of high-voltage faults, and the waveforms go crazy during the fault, which could lead to failure modes in the transfer switch which wouldn't be found on a "clean" disc…

That applies to other kinds of reliability testing too, especially everything that goes through a network. It's easy to handle when something is completely unavailable (i.e. instant error), but when something, be it a database or some endpoint, is available but horribly slow, that's a whole different thing.

Seems like we need systems designed with "suicide" mechanisms built in, so that if they detect that they have a poor quality of life (err, I mean, that they're providing a poor quality of service) they'll shut down completely.

Re: OVH outage explained

#66

Post by fulafel »

shrike wrote: post #39

When I was at AWS we were using generators from a large commercial supplier. We were constantly having issues with them refusing to take over if there wasn't sufficient load. Doing so puts lots of stress on a generator and can significantly shorten its life. We went to the manufacturer and tried to get them to make a firmware change; we wanted the generators to sacrifice themselves under most every circumstance (shor…

Why Diesel generators? Natural gas or gasoline should avoid wet stacking.

Re: OVH outage explained

#67

Post by Johnny555 »

robryk wrote: post #23
delsarto wrote: post #9

Earlier quoted context omitted.

I think they're just large, complicated physical things where stuff goes wrong, even if you're maintaining and testing. My best story was as a young student turning up for my helpdesk shift at about 5:50am to a phalanx of fire engines and the Hazmat team. The generator in the basement was maintained and tested. This time it had started when power went out, but there was a pump that filled a holding tank from a 10,000…

The particular failure mode you're talking about is reasonably easy to reliably remove: all fuel piping should be under lower than atmospheric pressure. You accomplish this by installing suction pumps next to consumers (and install additional pumps if the height difference is too large). That way, if a pipe breaks (except for the short pieces of pipe between the pipe and consumer), there will be no sustained fuel lea…

Building constraints don't always make it that easy -- the generator could be below the tank level, so siphon action can still siphon the tank empty, or it can be a few floors about the tank, too high to suck fuel from the tank so it has to be pushed up from below.

Re: OVH outage explained

#68

Post by etaioinshrdlu »

shrike wrote: post #39

When I was at AWS we were using generators from a large commercial supplier. We were constantly having issues with them refusing to take over if there wasn't sufficient load. Doing so puts lots of stress on a generator and can significantly shorten its life. We went to the manufacturer and tried to get them to make a firmware change; we wanted the generators to sacrifice themselves under most every circumstance (shor…

It seems like excessively low load would be a relatively easy problem to solve. One could heat tanks of water to boiling and vent the steam, that could probably absorb as much energy as needed.

Or just crank up the AC :)

Re: OVH outage explained

#69

Post by tbrownaw »

gech wrote: post #60
grecy wrote: post #49

Earlier quoted context omitted.

> This'd waste a ton of fuel, though, enough so that I'm not actually sure whether it'd be cheaper than eating an engine. I think you'll find fuel is extremely cheap, even when you're burning many gallons per minute.

There's the externality of pollution you didn't consider.

Since the generators apparently run less efficiently at partial load, that's not a given either. They'll almost certainly produce more pollution per unit fuel. Hm, is that a few percent more, or a few times more?

Re: OVH outage explained

#70

Post by userbinator »

fulafel wrote: post #66
shrike wrote: post #39

When I was at AWS we were using generators from a large commercial supplier. We were constantly having issues with them refusing to take over if there wasn't sufficient load. Doing so puts lots of stress on a generator and can significantly shorten its life. We went to the manufacturer and tried to get them to make a firmware change; we wanted the generators to sacrifice themselves under most every circumstance (shor…

Why Diesel generators? Natural gas or gasoline should avoid wet stacking.

Diesel tends to be used more because they tend to be much lower maintenance than other fuel engines, and the cost of the fuel itself is also lower due to the higher efficiency.
Post reply on HN