Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OVH outage explained

status.ovh.net

Post reply on HN
Print view View original

31–40 of 129 posts

Re: OVH outage explained

#31

Post by specialp »

Xylakant wrote: post #6
duncanawoods wrote: post #4

Why are backup generators so unreliable? And that is unreliable by the standard of a non-critical system let alone that of an emergency system where reliability is their sole purpose. Is it just a (non)-survivorship bias where we most commonly talk about the failure cases and they actually have a stellar 99.99999% record that doesn't make headlines?

Most of the time, it's not the actual generator that fails. It's often either the part that detects power failure (fluctuations, brownouts, ... are harder to detect than total failure) or the switchover hardware to the generator. If you want to switch 20KV, you're really handling an awful lot of energy. Physics kinda get in the way there. Switching paket based networks is often much easier because you can just hold o…

We've had this happen in our small onsite datacenter. A single phase out of the 3 changed waveform after a lightning strike on a utility pole. The power monitor to turn on the generator didn't see this a problem, but the UPS did. So the UPS was on for several minutes and I noticed some of the ceiling lights out that depended on that phase. Fortunately I and another employee were there and turned on the manual transfer to put us on generator power.

Disaster backups are hard to test as to do it properly they have to be integration tested, and doing so might break production. Sure your generator might power on for 5 minutes a month and make people happy, but there are many unforeseen things that are hard to test for.

Re: OVH outage explained

#32

Post by jedisct1 »

folbec wrote: post #13
praseodym wrote: post #12

Earlier quoted context omitted.

Details about the RBX outage were already posted earlier: http://travaux.ovh.net/?do=details&id=28244#comment35217

According to their maintenance postings, RBX was an upgrade problem to the software driver of the optical fiber : "RBX: We had a problem on the optical network that allows RBX to be connected with the interconnection points we have in Paris, Frankfurt, Amsterdam, London, Brussels. The origin of the problem is a software bug on the optical equipment, which caused the configuration to be lost and the connection to be c…

They gave more details in the French version. The equipment is Cisco NCS2000.

Re: OVH outage explained

#33

Post by pmontra »

ngrilly wrote: post #19

I have a hard time understanding why they needed humans babysitting the restart of servers and services. Servers and services are supposed to automatically restart after the power came back. This implies OVH doesn't regularly test server restarts with something like Netflix Chaos Monkey. Moreover, the power lines were not really redundant in SBG, and regarding the network downtime in RBX, it looks like the network co…

I have some VPSes there. All of them run independent services, no redundancy.

They can restart all the servers they want if they manage not to kill any of my processes, break network connections, change IP addresses. If they can't (it's probably hard) it's up to me to implement a reduntant architecture. Then maybe I'll also implement some random kill switch to proof that the system is resilient to failures.

Re: OVH outage explained

#34

Post by cesarb »

I wonder how easy it is to really test these failover systems. Even if you disconnect the external power (which is risky, since you lose redundancy), it will be a "clean" disconnection, with the power going neatly to zero. I've found a few post-mortems of high-voltage faults, and the waveforms go crazy during the fault, which could lead to failure modes in the transfer switch which wouldn't be found on a "clean" disconnection.

Re: OVH outage explained

#35

Post by vbernat »

ngrilly wrote: post #19

I have a hard time understanding why they needed humans babysitting the restart of servers and services. Servers and services are supposed to automatically restart after the power came back. This implies OVH doesn't regularly test server restarts with something like Netflix Chaos Monkey. Moreover, the power lines were not really redundant in SBG, and regarding the network downtime in RBX, it looks like the network co…

Restarting a whole datacenter from scratch is different from restarting individual servers. If a service needs another service running on another server, it may fail to start. For example, many servers may need the DHCP server. Some servers may need the DNS. Some others may need Kafka to be up and running, etc. You may have some documentation on which servers to start first, but things are moving so fast that it's not unlikely you'll need to rely on supervision to know which service was able to start correctly.

Re: OVH outage explained

#36

Post by Cyph0n »

kyledrake wrote: post #30

My datacenter has 2 power channels, coming from two different power grids. They have never had a full outage, even when a substation two blocks away started on fire. Even if we lost power on one power channel, we would still be operational. Even if the backup power failed. That's the reason you use redundant power. If your "cloud provider" is charging you a lot but not providing your server redundant power from two g…

The power grid in and of itself is a highly interconnected and redundant system. The fact that a substation fire didn't bring down your datacenter doesn't necessarily mean that the "redundant" setup was responsible.

Unless the two grids are fully isolated (e.g., from two different countries), there is always going to be some overlap in the two power sources.

Re: OVH outage explained

#37

Post by gedrap »

cesarb wrote: post #34

I wonder how easy it is to really test these failover systems. Even if you disconnect the external power (which is risky, since you lose redundancy), it will be a "clean" disconnection, with the power going neatly to zero. I've found a few post-mortems of high-voltage faults, and the waveforms go crazy during the fault, which could lead to failure modes in the transfer switch which wouldn't be found on a "clean" disc…

That applies to other kinds of reliability testing too, especially everything that goes through a network.

It's easy to handle when something is completely unavailable (i.e. instant error), but when something, be it a database or some endpoint, is available but horribly slow, that's a whole different thing.

Re: OVH outage explained

#38

Post by jijji »

Every small datacenter has a fake diesel generator that doesn't work. I worked for one datacenter and they purposely bought a non-working diesel generator and moved it into a room with the sole purpose to pitch it to investors that they had a working secondary power source.

Re: OVH outage explained

#39

Post by shrike »

When I was at AWS we were using generators from a large commercial supplier. We were constantly having issues with them refusing to take over if there wasn't sufficient load. Doing so puts lots of stress on a generator and can significantly shorten its life.

We went to the manufacturer and tried to get them to make a firmware change; we wanted the generators to sacrifice themselves under most every circumstance (short of danger to humans). Generators are an irrelevant cost compared to the cost of an outage. The manufacturer refused, even when we offered to buy them without warranties.

I don't know if they still do but at that point AWS started to buy basically the same generators straight from China and writing their own firmware.

There is also a great story about an AWS authored firmware update on a set of louver microcontrollers causing a partial outage.

AWS really likes to own the entire stack.

*It has been years since I left, details are a bit fuzzy but I think I've got these right.

Re: OVH outage explained

#40

Post by walshemj »

shrike wrote: post #39

When I was at AWS we were using generators from a large commercial supplier. We were constantly having issues with them refusing to take over if there wasn't sufficient load. Doing so puts lots of stress on a generator and can significantly shorten its life. We went to the manufacturer and tried to get them to make a firmware change; we wanted the generators to sacrifice themselves under most every circumstance (shor…

And at AWS scale you can always have a human team present to provide a human back up the cost of having 2 or 3 people on an overnight shift is trivial compared to the down time costs.
Post reply on HN