Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OVH outage explained

status.ovh.net

Post reply on HN
Print view View original

71–80 of 129 posts

Re: OVH outage explained

#71

Post by joelhaasnoot »

saulrh wrote: post #56
grecy wrote: post #49

Earlier quoted context omitted.

> This'd waste a ton of fuel, though, enough so that I'm not actually sure whether it'd be cheaper than eating an engine. I think you'll find fuel is extremely cheap, even when you're burning many gallons per minute.

Hmmm. I didn't actually do the numbers. Let's see... A randomly-googled 2MW diesel generator consumes 50 gallons/hour at quarter load and 160 gallons/hour at full load [1]. So let's say it's 100 gallons/hour to run a 2 MW generator at full load instead of quarter load. The OVH incident report [OP] says that their data center has two cables in each carrying 10 MVA (mega-volts-amperes, ~ watts), giving us 20 MW as roug…

Fuel is much more expensive in Europe. A liter of Diesel is about 1.40, so $6.50 per gallon.

Re: OVH outage explained

#72

Post by jankey »

etaioinshrdlu wrote: post #68
shrike wrote: post #39

When I was at AWS we were using generators from a large commercial supplier. We were constantly having issues with them refusing to take over if there wasn't sufficient load. Doing so puts lots of stress on a generator and can significantly shorten its life. We went to the manufacturer and tried to get them to make a firmware change; we wanted the generators to sacrifice themselves under most every circumstance (shor…

It seems like excessively low load would be a relatively easy problem to solve. One could heat tanks of water to boiling and vent the steam, that could probably absorb as much energy as needed. Or just crank up the AC :)

Huge DCs like Amazons and ours at Google are very efficient, cooling uses at most 10% of total power. So cranking up this wouldn't help. Nor wouldn't it be possible usual anyway, as these are typically using evaporation cooling, which can't really be cranked up like traditional compression cooling. And keeping a huge tank and heater ready would introduce another point of failure. The easiest is just to have your servers do some heavy calculations.

Re: OVH outage explained

#73

Post by snom380 »

ngrilly wrote: post #61
vbernat wrote: post #35

Earlier quoted context omitted.

Restarting a whole datacenter from scratch is different from restarting individual servers. If a service needs another service running on another server, it may fail to start. For example, many servers may need the DHCP server. Some servers may need the DNS. Some others may need Kafka to be up and running, etc. You may have some documentation on which servers to start first, but things are moving so fast that it's no…

I don't buy this argument. If a service needs another service running on another server which is not already available, it should retry with an exponential backoff. Doing anything else is a bug.

I agree it's a bug, but this is far, far more difficult to test against with a chaos monkey than one might think. With a whole datacenter down, not to speak of multiple availability zones, problems and dependencies will crop up that are extremely difficult to anticipate. Maybe you expected one other AZ to be up so you could provision servers from there, but now you have to somehow get one of them bootstrapped. Or maybe during the outage power was cycled on some equipment one too many times, causing some failsafe to trip, or maybe some of the core infrastructure like your switches and routers expect some core service to be up, but that core service can't get online without working network.

Re: OVH outage explained

#74

Post by jankey »

saulrh wrote: post #46
melq wrote: post #41

Earlier quoted context omitted.

Do you know why its harder for a generator to power an unloaded circuit? Just curious.

The generator's mechanical parts will perform more efficiently at some speeds/loads than at others. Everything happening at a different temperature and pressure will change fluid flows and forces and put different stresses in different places. Combustion won't happen cleanly, oil and gases won't flow the way they're designed to and the engine will get dirty, etc. If you're running outside of the region the engine was…

Dummy load looks like this: http://www.simplexdirect.com/images/pic-l-atlasTrailer.jpg It's "only" a megawatt but there are bigger ones as well.

Basically a big hairdryer. :-)

Re: OVH outage explained

#75

Post by TheSpiceIsLife »

wrigby wrote: post #47
melq wrote: post #41

Earlier quoted context omitted.

Do you know why its harder for a generator to power an unloaded circuit? Just curious.

My understanding is that it's the diesel engines that don't like it (the generator head itself is fine). Basically, some of the fuel in the combustion chamber doesn't get burned, and comes out the exhaust. This builds up in the exhaust system, forming an ooze in the pipes. I think it's worse on a turbo diesel, because now the ooze is coming into the turbocharger. https://en.wikipedia.org/wiki/Wet_stacking

Adding a heat-bank / load-bank[1] solves this problem.

1. Typically these are just big fan heaters https://en.m.wikipedia.org/wiki/Load_bank

Re: OVH outage explained

#76

Post by kittens321 »

shrike wrote: post #39

When I was at AWS we were using generators from a large commercial supplier. We were constantly having issues with them refusing to take over if there wasn't sufficient load. Doing so puts lots of stress on a generator and can significantly shorten its life. We went to the manufacturer and tried to get them to make a firmware change; we wanted the generators to sacrifice themselves under most every circumstance (shor…

I'm not an expert, but a high power device safe load means that it won't place a lot of the load on itself. when it places the load on itself it gets really hot, so hot that parts may melt, and things that were supposed to be insulated may now be conductive.

Doing high voltage electronics without a license is a crime in a lot of places. I'm pretty sure a person with the right specialty can add a load so you won't need to hack up the firmware to violate its safe operating ranges.

If someone told me this in person I'd consider contacting authorities over their negligent behaviour. The manufacturer was willing to lose business for a safety matter even though allowing the firmware hacks would be really easy.

Re: OVH outage explained

#77

Post by TheSpiceIsLife »

userbinator wrote: post #70
fulafel wrote: post #66

Earlier quoted context omitted.

Why Diesel generators? Natural gas or gasoline should avoid wet stacking.

Diesel tends to be used more because they tend to be much lower maintenance than other fuel engines, and the cost of the fuel itself is also lower due to the higher efficiency.

I think also storing diesel is easier. It's not very flammable as compared petrol (gasoline) or LPG / LNG.

Although diesel can get microbial contamination if not used in 6 - 12 months.[1]

1. https://en.m.wikipedia.org/wiki/Microbial_contamination_of_d...

Re: OVH outage explained

#78

Post by ngrilly »

snom380 wrote: post #73
ngrilly wrote: post #61

Earlier quoted context omitted.

I don't buy this argument. If a service needs another service running on another server which is not already available, it should retry with an exponential backoff. Doing anything else is a bug.

I agree it's a bug, but this is far, far more difficult to test against with a chaos monkey than one might think. With a whole datacenter down, not to speak of multiple availability zones, problems and dependencies will crop up that are extremely difficult to anticipate. Maybe you expected one other AZ to be up so you could provision servers from there, but now you have to somehow get one of them bootstrapped. Or may…

Thanks for the great explanation. I agree that a Chaos Monkey test something a lot more easier to recover from than a whole datacenter getting down.

I remember having read somewhere about some company (Facebook/Google/Amazon/Twitter/Dropbox or something at the same scale) that regularly simulates a whole datacenter failure, which made believe it is possible to automatically recover from this.

Are you saying that even the companies I mentioned have the same issues as OVH when they recover from a complete power failure?

Re: OVH outage explained

#79

Post by robocat »

robryk wrote: post #23
delsarto wrote: post #9

Earlier quoted context omitted.

I think they're just large, complicated physical things where stuff goes wrong, even if you're maintaining and testing. My best story was as a young student turning up for my helpdesk shift at about 5:50am to a phalanx of fire engines and the Hazmat team. The generator in the basement was maintained and tested. This time it had started when power went out, but there was a pump that filled a holding tank from a 10,000…

The particular failure mode you're talking about is reasonably easy to reliably remove: all fuel piping should be under lower than atmospheric pressure. You accomplish this by installing suction pumps next to consumers (and install additional pumps if the height difference is too large). That way, if a pipe breaks (except for the short pieces of pipe between the pipe and consumer), there will be no sustained fuel lea…

So now the system fails due to frothy diesel. Or due to failures of the additional pumps (adding mechanical complexity to a system often decreases overall reliability)

Re: OVH outage explained

#80

Post by jcrawfordor »

etaioinshrdlu wrote: post #68
shrike wrote: post #39

When I was at AWS we were using generators from a large commercial supplier. We were constantly having issues with them refusing to take over if there wasn't sufficient load. Doing so puts lots of stress on a generator and can significantly shorten its life. We went to the manufacturer and tried to get them to make a firmware change; we wanted the generators to sacrifice themselves under most every circumstance (shor…

It seems like excessively low load would be a relatively easy problem to solve. One could heat tanks of water to boiling and vent the steam, that could probably absorb as much energy as needed. Or just crank up the AC :)

There are load banks made for generators, just large resistive grids. A facility that I work at has one so that we can exercise the generator at operating load without actually switching over our power. So I would think a good solution for a DC might be to have load banks you could switch in when necessary.
Post reply on HN