Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OVH outage explained

status.ovh.net

Post reply on HN
Print view View original

121–129 of 129 posts

Re: OVH outage explained

#121

Post by cup-of-tea »

gnu8 wrote: post #86
shrike wrote: post #39

When I was at AWS we were using generators from a large commercial supplier. We were constantly having issues with them refusing to take over if there wasn't sufficient load. Doing so puts lots of stress on a generator and can significantly shorten its life. We went to the manufacturer and tried to get them to make a firmware change; we wanted the generators to sacrifice themselves under most every circumstance (shor…

The company that maintains my generators has a dummy load that is basically a large resistor bank and fan on a trailer. It seems like Amazon could just build something like that on a larger scale to load the generators - short term they could even rent the dummy load trailers and hook them up in the parking lot, right? For a permanent solution, have logic to switch in the appropriate amount of resistance to maintain…

Can't you just have a smaller generator for low load? Literally wasting fuel so you can run a large generator for a low load should be criminal.

Re: OVH outage explained

#122

Post by dx034 »

walshemj wrote: post #40
shrike wrote: post #39

When I was at AWS we were using generators from a large commercial supplier. We were constantly having issues with them refusing to take over if there wasn't sufficient load. Doing so puts lots of stress on a generator and can significantly shorten its life. We went to the manufacturer and tried to get them to make a firmware change; we wanted the generators to sacrifice themselves under most every circumstance (shor…

And at AWS scale you can always have a human team present to provide a human back up the cost of having 2 or 3 people on an overnight shift is trivial compared to the down time costs.

I wonder why they couldn't do that at OVH. The data centres are always manned and they had 8 minutes on batteries. Why didn't they detect after 20 seconds that generators don't start despite power failure and triggered a manual failover? Sounds like something the staff should be able to do?

Re: OVH outage explained

#123

Post by dx034 »

tyingq wrote: post #11
cjg_ wrote: post #5

Earlier quoted context omitted.

In this case the generators themselves worked, just not the mechanism to cut over to them.

Generally called a "static switch", and they are prone to issues. Here's one: https://www.aegps.com/en/products/ups-accessories/static-tra...

Are they automatically a single point of failure or can you install several of those on one site?

Re: OVH outage explained

#124

Post by dx034 »

jedisct1 wrote: post #32
folbec wrote: post #13

Earlier quoted context omitted.

According to their maintenance postings, RBX was an upgrade problem to the software driver of the optical fiber : "RBX: We had a problem on the optical network that allows RBX to be connected with the interconnection points we have in Paris, Frankfurt, Amsterdam, London, Brussels. The origin of the problem is a software bug on the optical equipment, which caused the configuration to be lost and the connection to be c…

They gave more details in the French version. The equipment is Cisco NCS2000.

Let's hope that the second equipment they ordered is not Cisco so that they have full redundancy.

Re: OVH outage explained

#125

Post by joelhaasnoot »

gaadd33 wrote: post #83
joelhaasnoot wrote: post #71

Earlier quoted context omitted.

Fuel is much more expensive in Europe. A liter of Diesel is about 1.40, so $6.50 per gallon.

Does Europe tax diesel the same way no matter where it's used? I know in the US diesel for generators is exempt from a lot of taxes resulting in it being significantly cheaper in some states.

You're right, some countries have exceptions. Netherlands where I live used to, but no longer does. Not sure about France.

Re: OVH outage explained

#126

Post by gnu8 »

cup-of-tea wrote: post #121
gnu8 wrote: post #86

Earlier quoted context omitted.

The company that maintains my generators has a dummy load that is basically a large resistor bank and fan on a trailer. It seems like Amazon could just build something like that on a larger scale to load the generators - short term they could even rent the dummy load trailers and hook them up in the parking lot, right? For a permanent solution, have logic to switch in the appropriate amount of resistance to maintain…

Can't you just have a smaller generator for low load? Literally wasting fuel so you can run a large generator for a low load should be criminal.

You might not realize how much fuel it takes just to maintain generators. They have to be exercised, load tested, overhauled, all on top of whatever time they spend providing backup power. Which is just as well, because an engine that sits without ever running won't operate when it is most needed, and the fuel can stagnate, or even attract water and grow algae.

Re: OVH outage explained

#127

Post by gnu8 »

tgtweak wrote: post #117
gnu8 wrote: post #86

Earlier quoted context omitted.

The company that maintains my generators has a dummy load that is basically a large resistor bank and fan on a trailer. It seems like Amazon could just build something like that on a larger scale to load the generators - short term they could even rent the dummy load trailers and hook them up in the parking lot, right? For a permanent solution, have logic to switch in the appropriate amount of resistance to maintain…

There are, unfortunately, a lot of things between on and off when it comes to utility power. Over voltage, voltage drop (which can cause fuses to blow since amperage goes up), phase outage, phase imbalance, harmonic noise (causing ground/neutral feedback), micro outages (few cycles; Most downtime I've seen in datacenters has been due to irregular power, not power loss. Case in point. Dummy loads are good for testing…

I would say the automatic transfer switch should be configured conservatively and go to backup power at the first hint of instability. If irregular power is the usual cause of downtime then I would expect the ATS to cope with those conditions, or it gets replaced with one that will.

Good point about the lack of variability. Not sure if the generators would take kindly to suddenly switching out a 1 mw dummy load to free capacity for real load.

On the other hand, it's not like these engines have fixed RPM throttles, and they should be able to handle a wide range of loads, albeit not at peak efficiency. Something doesn't fully make sense about that story. Maybe the generators were sized for their ultimate planned capacity and thus way too much for their deployment at the time.

Re: OVH outage explained

#128

Post by ADefenestrator »

gnu8 wrote: post #127
tgtweak wrote: post #117

Earlier quoted context omitted.

There are, unfortunately, a lot of things between on and off when it comes to utility power. Over voltage, voltage drop (which can cause fuses to blow since amperage goes up), phase outage, phase imbalance, harmonic noise (causing ground/neutral feedback), micro outages (few cycles; Most downtime I've seen in datacenters has been due to irregular power, not power loss. Case in point. Dummy loads are good for testing…

I would say the automatic transfer switch should be configured conservatively and go to backup power at the first hint of instability. If irregular power is the usual cause of downtime then I would expect the ATS to cope with those conditions, or it gets replaced with one that will. Good point about the lack of variability. Not sure if the generators would take kindly to suddenly switching out a 1 mw dummy load to fr…

I thought large diesel generators generally did operate at a fixed speed in order to output 60Hz. I know inverter/generators are common these days for small gas gensets, but have the big diesel ones switched over as well?

Re: OVH outage explained

#129

Post by ADefenestrator »

fulafel wrote: post #66
shrike wrote: post #39

When I was at AWS we were using generators from a large commercial supplier. We were constantly having issues with them refusing to take over if there wasn't sufficient load. Doing so puts lots of stress on a generator and can significantly shorten its life. We went to the manufacturer and tried to get them to make a firmware change; we wanted the generators to sacrifice themselves under most every circumstance (shor…

Why Diesel generators? Natural gas or gasoline should avoid wet stacking.

Easier to generate a fixed-frequency AC output. Diesel engines can run at a fixed RPM and adjust fueling to match load, where gasoline and natural gas need a narrow AFR and have to vary rotation speed for power output. Maybe less of an issue these days as inverter/generators are more common, though I dunno if that's the case for the big generators.
Post reply on HN