Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OVH outage explained

status.ovh.net

Post reply on HN
Print view View original

11–20 of 129 posts

Re: OVH outage explained

#11

Post by tyingq »

cjg_ wrote: post #5
duncanawoods wrote: post #4

Why are backup generators so unreliable? And that is unreliable by the standard of a non-critical system let alone that of an emergency system where reliability is their sole purpose. Is it just a (non)-survivorship bias where we most commonly talk about the failure cases and they actually have a stellar 99.99999% record that doesn't make headlines?

In this case the generators themselves worked, just not the mechanism to cut over to them.

Generally called a "static switch", and they are prone to issues. Here's one: https://www.aegps.com/en/products/ups-accessories/static-tra...

Re: OVH outage explained

#13

Post by folbec »

praseodym wrote: post #12
jedisct1 wrote: post #2

Great summary. That's for the SBG outage, not the RBX one though.

Details about the RBX outage were already posted earlier: http://travaux.ovh.net/?do=details&id=28244#comment35217

According to their maintenance postings, RBX was an upgrade problem to the software driver of the optical fiber :

"RBX: We had a problem on the optical network that allows RBX to be connected with the interconnection points we have in Paris, Frankfurt, Amsterdam, London, Brussels. The origin of the problem is a software bug on the optical equipment, which caused the configuration to be lost and the connection to be cut from our site in RBX. We handed over the backup of the software configuration as soon as we diagnosed the source of the problem and the DC can be reached again. The incident on RBX is fixed. With the manufacturer, we are looking for the origin of the software bug and also looking to avoid this kind of critical incident. "

Re: OVH outage explained

#16

Post by [deleted] »

duncanawoods wrote: post #4

Why are backup generators so unreliable? And that is unreliable by the standard of a non-critical system let alone that of an emergency system where reliability is their sole purpose. Is it just a (non)-survivorship bias where we most commonly talk about the failure cases and they actually have a stellar 99.99999% record that doesn't make headlines?

[deleted]

Re: OVH outage explained

#17

Post by bringtheaction »

tyingq wrote: post #3

Very transparent, kudos for that. A reminder to everyone, though, that the brochure doesn't always match reality. OVH pitches itself as a global data center leader, with 20 facilities, 2 of them amongst the largest in the world. Not to harp on them, just to say that having some backup plan on a separate provider is always a good idea, no matter who your primary is.

Also when you decide on a backup, make sure your backup is not hosting their servers in the same DCs as is your primary, or worse yet that your backup is renting their servers from the same people you rent your primary from.

Re: OVH outage explained

#18

Post by blattimwind »

Xylakant wrote: post #6
duncanawoods wrote: post #4

Why are backup generators so unreliable? And that is unreliable by the standard of a non-critical system let alone that of an emergency system where reliability is their sole purpose. Is it just a (non)-survivorship bias where we most commonly talk about the failure cases and they actually have a stellar 99.99999% record that doesn't make headlines?

Most of the time, it's not the actual generator that fails. It's often either the part that detects power failure (fluctuations, brownouts, ... are harder to detect than total failure) or the switchover hardware to the generator. If you want to switch 20KV, you're really handling an awful lot of energy. Physics kinda get in the way there. Switching paket based networks is often much easier because you can just hold o…

To add to this, backup power systems are in fairly widespread deployment and bigger installations are tested and maintained on a schedule. You'll never see "hospital transferred emergency-available circuits to backup system; no malfunction" in the news, yet it happens every few weeks in every installation.

Generators themselves are very reliable machines. The engines are cast iron (instead of cast aluminium as found in all cars since two decades or so) and are nowhere near as close to the edge of material capacity compared to car engines. Electrical generators are essentially just a big blob of metal, they are simple, efficient and very reliable technology giving many decades of service. Generator controllers are designed to be reliable and aren't terribly complicated, either.

Re: OVH outage explained

#19

Post by ngrilly »

I have a hard time understanding why they needed humans babysitting the restart of servers and services. Servers and services are supposed to automatically restart after the power came back. This implies OVH doesn't regularly test server restarts with something like Netflix Chaos Monkey.

Moreover, the power lines were not really redundant in SBG, and regarding the network downtime in RBX, it looks like the network core is not really redundant either.

I'd be curious to know how competitors like AWS and Google Cloud mitigate this kind of problems.

Re: OVH outage explained

#20

Post by icebraining »

ngrilly wrote: post #19

I have a hard time understanding why they needed humans babysitting the restart of servers and services. Servers and services are supposed to automatically restart after the power came back. This implies OVH doesn't regularly test server restarts with something like Netflix Chaos Monkey. Moreover, the power lines were not really redundant in SBG, and regarding the network downtime in RBX, it looks like the network co…

OVH runs servers for their clients, do you really expect them to randomly restart their clients' servers? Netflix is not a hosting provider, the situations are not comparable.
Post reply on HN