Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OVH outage explained

status.ovh.net

Post reply on HN
Print view View original

41–50 of 129 posts

Re: OVH outage explained

#41

Post by melq »

shrike wrote: post #39

When I was at AWS we were using generators from a large commercial supplier. We were constantly having issues with them refusing to take over if there wasn't sufficient load. Doing so puts lots of stress on a generator and can significantly shorten its life. We went to the manufacturer and tried to get them to make a firmware change; we wanted the generators to sacrifice themselves under most every circumstance (shor…

Do you know why its harder for a generator to power an unloaded circuit? Just curious.

Re: OVH outage explained

#42

Post by rmdoss »

I have been a big OVH advocate (cheap prices + free DDoS protection + good hardware) and have been using them for years.

However, I am really frustrated by the lack of communication during this incident. All I had was the tweets from the founder, while their status page was down and I had no other place to reach out to them (phone + ticket system down).

What is worse is that they treat the individual SBGs (SBG1, SBG2, etc) as individual availability zones, when they truly are not. I had an application using their vrack on different SBGs availability zones and they all went down.

Anyway, glad they are fixing it, and glad for the clarification that they don't have real availability zones. Will design my applications better next time.

Re: OVH outage explained

#43

Post by [deleted] »

melq wrote: post #41
shrike wrote: post #39

When I was at AWS we were using generators from a large commercial supplier. We were constantly having issues with them refusing to take over if there wasn't sufficient load. Doing so puts lots of stress on a generator and can significantly shorten its life. We went to the manufacturer and tried to get them to make a firmware change; we wanted the generators to sacrifice themselves under most every circumstance (shor…

Do you know why its harder for a generator to power an unloaded circuit? Just curious.

[deleted]

Re: OVH outage explained

#44

Post by laumars »

jijji wrote: post #38

Every small datacenter has a fake diesel generator that doesn't work. I worked for one datacenter and they purposely bought a non-working diesel generator and moved it into a room with the sole purpose to pitch it to investors that they had a working secondary power source.

"Every"? I know for a fact that we don't because we've fallen back to the redundant generator before and it's worked flawlessly. (pity the same couldn't have been said for some of our other redundant systems).

Re: OVH outage explained

#45

Post by lima »

melq wrote: post #41
shrike wrote: post #39

When I was at AWS we were using generators from a large commercial supplier. We were constantly having issues with them refusing to take over if there wasn't sufficient load. Doing so puts lots of stress on a generator and can significantly shorten its life. We went to the manufacturer and tried to get them to make a firmware change; we wanted the generators to sacrifice themselves under most every circumstance (shor…

Do you know why its harder for a generator to power an unloaded circuit? Just curious.

As far as I understood, it's mostly because it won't reach optimal temperatures and pressure, which then leads to incomplete combustion and carbon buildups in the cylinders.

Re: OVH outage explained

#46

Post by saulrh »

melq wrote: post #41
shrike wrote: post #39

When I was at AWS we were using generators from a large commercial supplier. We were constantly having issues with them refusing to take over if there wasn't sufficient load. Doing so puts lots of stress on a generator and can significantly shorten its life. We went to the manufacturer and tried to get them to make a firmware change; we wanted the generators to sacrifice themselves under most every circumstance (shor…

Do you know why its harder for a generator to power an unloaded circuit? Just curious.

The generator's mechanical parts will perform more efficiently at some speeds/loads than at others. Everything happening at a different temperature and pressure will change fluid flows and forces and put different stresses in different places. Combustion won't happen cleanly, oil and gases won't flow the way they're designed to and the engine will get dirty, etc. If you're running outside of the region the engine was designed to operate in you're stressing it in ways it was not designed to handle, lowering efficiency and shortening its lifespan.

(You could run the generator at its favorite speed and sink excess power into a dummy load. This'd waste a ton of fuel, though, enough so that I'm not actually sure whether it'd be cheaper than eating an engine. That and I'm not sure what a data-center-sized dummy load would look like.)

https://en.wikipedia.org/wiki/Power_band

Re: OVH outage explained

#47

Post by wrigby »

melq wrote: post #41
shrike wrote: post #39

When I was at AWS we were using generators from a large commercial supplier. We were constantly having issues with them refusing to take over if there wasn't sufficient load. Doing so puts lots of stress on a generator and can significantly shorten its life. We went to the manufacturer and tried to get them to make a firmware change; we wanted the generators to sacrifice themselves under most every circumstance (shor…

Do you know why its harder for a generator to power an unloaded circuit? Just curious.

My understanding is that it's the diesel engines that don't like it (the generator head itself is fine). Basically, some of the fuel in the combustion chamber doesn't get burned, and comes out the exhaust. This builds up in the exhaust system, forming an ooze in the pipes. I think it's worse on a turbo diesel, because now the ooze is coming into the turbocharger.

https://en.wikipedia.org/wiki/Wet_stacking

Re: OVH outage explained

#49

Post by grecy »

saulrh wrote: post #46
melq wrote: post #41

Earlier quoted context omitted.

Do you know why its harder for a generator to power an unloaded circuit? Just curious.

The generator's mechanical parts will perform more efficiently at some speeds/loads than at others. Everything happening at a different temperature and pressure will change fluid flows and forces and put different stresses in different places. Combustion won't happen cleanly, oil and gases won't flow the way they're designed to and the engine will get dirty, etc. If you're running outside of the region the engine was…

> This'd waste a ton of fuel, though, enough so that I'm not actually sure whether it'd be cheaper than eating an engine.

I think you'll find fuel is extremely cheap, even when you're burning many gallons per minute.

Re: OVH outage explained

#50

Post by kyledrake »

Cyph0n wrote: post #36
kyledrake wrote: post #30

My datacenter has 2 power channels, coming from two different power grids. They have never had a full outage, even when a substation two blocks away started on fire. Even if we lost power on one power channel, we would still be operational. Even if the backup power failed. That's the reason you use redundant power. If your "cloud provider" is charging you a lot but not providing your server redundant power from two g…

The power grid in and of itself is a highly interconnected and redundant system. The fact that a substation fire didn't bring down your datacenter doesn't necessarily mean that the "redundant" setup was responsible. Unless the two grids are fully isolated (e.g., from two different countries), there is always going to be some overlap in the two power sources.

The trick to building good datacenters is you make as many things redundant as possible, and it reduces the chance of one thing taking the entire datacenter down. Being connected to two external power providers is not the reason our datacenter stayed up, it's one of the many carefully designed redundancies that all add up to higher availability.

When you have an A/B isolated power datacenter, you can (and my datacenter regularly does) shut off each channel to test that the backup systems are working, without worrying that a failed test will bring down literally every customer in the datacenter. That testing would have likely caught this problem at OVH if they were able to do it, but they instead decided to just toss all the power into a single channel, making it impossible to do testing like this without a decent chance of widespread outages. So they just had to assume it would work, until it didn't.

Because our power channels are isolated, there's almost no difference between turning each one off separately, and then turning both of them off at the same time. So if we were to lose external power on both (which would be a history books blackout for how this DC gets its power, including an almost direct connection to a hydro-electric dam), you would have two well-tested backup power channels ready to go. Nothing is ever a guarantee, but I like my chances.

Post reply on HN