OVH outage explained
status.ovh.net
OVH outage explained
1–10 of 129 posts
Re: OVH outage explained
#2Great summary. That's for the SBG outage, not the RBX one though.
Re: OVH outage explained
#3Very transparent, kudos for that. A reminder to everyone, though, that the brochure doesn't always match reality. OVH pitches itself as a global data center leader, with 20 facilities, 2 of them amongst the largest in the world. Not to harp on them, just to say that having some backup plan on a separate provider is always a good idea, no matter who your primary is.
Re: OVH outage explained
#4Why are backup generators so unreliable? And that is unreliable by the standard of a non-critical system let alone that of an emergency system where reliability is their sole purpose.
Is it just a (non)-survivorship bias where we most commonly talk about the failure cases and they actually have a stellar 99.99999% record that doesn't make headlines?
Re: OVH outage explained
#5Why are backup generators so unreliable? And that is unreliable by the standard of a non-critical system let alone that of an emergency system where reliability is their sole purpose. Is it just a (non)-survivorship bias where we most commonly talk about the failure cases and they actually have a stellar 99.99999% record that doesn't make headlines?
In this case the generators themselves worked, just not the mechanism to cut over to them.
Re: OVH outage explained
#6Why are backup generators so unreliable? And that is unreliable by the standard of a non-critical system let alone that of an emergency system where reliability is their sole purpose. Is it just a (non)-survivorship bias where we most commonly talk about the failure cases and they actually have a stellar 99.99999% record that doesn't make headlines?
Most of the time, it's not the actual generator that fails. It's often either the part that detects power failure (fluctuations, brownouts, ... are harder to detect than total failure) or the switchover hardware to the generator. If you want to switch 20KV, you're really handling an awful lot of energy. Physics kinda get in the way there.
Switching paket based networks is often much easier because you can just hold on to the packet and retransmit if a line fails, that's why it's much rarer to have total network failures.
Re: OVH outage explained
#7Why are backup generators so unreliable? And that is unreliable by the standard of a non-critical system let alone that of an emergency system where reliability is their sole purpose. Is it just a (non)-survivorship bias where we most commonly talk about the failure cases and they actually have a stellar 99.99999% record that doesn't make headlines?
Similar to many of us make backups in IT but how many actually do try them out and restore?
Re: OVH outage explained
#8Why are backup generators so unreliable? And that is unreliable by the standard of a non-critical system let alone that of an emergency system where reliability is their sole purpose. Is it just a (non)-survivorship bias where we most commonly talk about the failure cases and they actually have a stellar 99.99999% record that doesn't make headlines?
In this case the generators themselves worked, just not the mechanism to cut over to them.
I include that bit of the system in "backup generator" to distinguish it from "generator".
Re: OVH outage explained
#9Why are backup generators so unreliable? And that is unreliable by the standard of a non-critical system let alone that of an emergency system where reliability is their sole purpose. Is it just a (non)-survivorship bias where we most commonly talk about the failure cases and they actually have a stellar 99.99999% record that doesn't make headlines?
I think they're just large, complicated physical things where stuff goes wrong, even if you're maintaining and testing.
My best story was as a young student turning up for my helpdesk shift at about 5:50am to a phalanx of fire engines and the Hazmat team.
The generator in the basement was maintained and tested. This time it had started when power went out, but there was a pump that filled a holding tank from a 10,000ish litre tank under the building (speced to run for a week apparently). A pipe had cracked, and so it just kept pumping fuel on to the floor, carpark and into the local stream ...
The only person in the building overnight was the operator (tape changer and report runner) some 5 stories above all this, who eventually smelt it and eventually investgated. It was a massive nightmare, and the place stank for weeks.
Then there was the time the halon system went off unnecessarily...
Re: OVH outage explained
#10Why are backup generators so unreliable? And that is unreliable by the standard of a non-critical system let alone that of an emergency system where reliability is their sole purpose. Is it just a (non)-survivorship bias where we most commonly talk about the failure cases and they actually have a stellar 99.99999% record that doesn't make headlines?
I think they're just large, complicated physical things where stuff goes wrong, even if you're maintaining and testing. My best story was as a young student turning up for my helpdesk shift at about 5:50am to a phalanx of fire engines and the Hazmat team. The generator in the basement was maintained and tested. This time it had started when power went out, but there was a pump that filled a holding tank from a 10,000…
I have a similar story of a smallish colocation site located in the cellar of a high school building in Switzerland where we hosted two servers for a small company ten years ago. The operator company had a generator at the site. During a thunderstorm there was a power outage, so at first all was fine. However a lightning nearby destroyed some critical circuitry of the generator. I additionally made the mistake of putting the watchdog system in the same colocation so that I didn't even get a notification about the outage. But after all I would have been powerless anyway.