Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OVH Incident in Strasbourg

status.ovh.com

Post reply on HN
Print view View original

181–190 of 207 posts

Re: OVH Incident in Strasbourg

#181

Post by peterwwillis »

jedisct1 wrote: post #179

Details here: http://travaux.ovh.net/?do=details&id=28244 Apparently, the root cause of that issue is a critical software bug in Cisco NCS 2000 transponders.

> "Diagnosis: All the transponder cards we use, ncs2k-400g-lk9, ncs2k-200g-cklc, are in "standby" state. One of the possible origins of such a state is the loss of configuration. So we recovered the backup and put back the configuration, which allowed the system to reconfigure all the transponder cards."

Their interfaces lost their configuration, and they re-applied configuration, and state came back. This does not equal critical software bug.

> "One of the solutions is to create 2 optical node systems instead of one. 2 systems, that means 2 databases and so in case of loss of configuration, only one system is down. If 50% of the links go through one of the systems, today we would have lost 50% of the capacity but not 100% of links."

This is a crap mitigation. They're still depending on the same hardware and process that led to the first outage, only now there's more of it, so there's more chances to fail.

If they had continuous configuration automation they would have detected when the router's state changed, identified the missing bits, and applied configuration.

"New" routers (as in, since 2011) have APIs and can even run code directly on the router in order to fulfill these requirements. Cisco has multiple white papers, and even provides complete products to manage and certify configuration is applied as desired, even in cloud-agnostic multi-tier networks. Even on old routers, practically all config management solutions out there have plugins to manage Cisco routers.

It's also ridiculous that they had no access to remote hands. This is IT 101.

Re: OVH Incident in Strasbourg

#182

Post by Gelob »

notyourday wrote: post #174
KeitIG wrote: post #18

I imagine Mr Good Guy at OVH telling some others: "guys we have a single point of failure in our architecture with SBG, maybe we should... - naaah it's fine, we do not have time nor resources" Then shit happens. edit: I have no idea what is happening exactly, but OVH being what it is, it seems extremely weird that all datacenters "can" get down at the same time, and it looks like a serious architecture problem to me…

This happens all the time. Every single thing that you see in software development happens in network engineering and data center engineering, except that where in development in general senior people who write software are capable of at least guestimating complexities to provide a marginally unified front against the unreasonable expectations of execs, it is pretty much never the case in neteng or dcops as those tha…

Any decent sr. network engineer or architect should be able to design you a network and explain the pros and cons, risks, and future scalability.

If someone doesn't know why a failure occured and they can't find out then they aren't looking hard enough

Re: OVH Incident in Strasbourg

#183

Post by Piskvorrr »

dx034 wrote: post #167
zaarn wrote: post #95

Earlier quoted context omitted.

The problem is usually not "not reliable in cold" but rather, the generators are X years old and the temperature is now changing in Europe from "mostly warm" to "warm over the day and icecold in the night" and finally aiming for "icecold all day", which means any equipment exposed will go through rather severe temperature changes. While generators are usually able to handle this with sufficiently low failure risk, th…

It wasn't even freezing overnight at Strasbourg [1], I doubt that the cold could cause any effects at 5C. Maybe they were run once a month but never tested under load? 2x20kv lines failing will have put them at near maximum load immediately, perhaps they weren't designed for that. [1] https://www.accuweather.com/en/fr/strasbourg/131836/daily-we...

Improbable. Even in a mostly-normal office building, the monthly generator test consisted essentially of "cut the mains power and see that no impact is perceived as UPS, batteries, and autostarted generators bear the load, in their turn for ~2 hrs total."

Re: OVH Incident in Strasbourg

#185

Post by abiox »

mbrameld wrote: post #172
nolok wrote: post #152

Earlier quoted context omitted.

Calling OVH a "smaller hosting company" (which you did do indirectly) is rather funny.

The person you're replying to was relating their experiences at hosting companies that are smaller than OVH. How does that imply that OVH is a smaller hosting company?

to me it's ambiguous; it could be either of

> in smaller hosting companies [like ovh]

> in smaller hosting companies [than ovh]

Re: OVH Incident in Strasbourg

#186

Post by zyztem »

wiz21c wrote: post #104

Damn, every emergency power supply I have encountered (the big ones with fuel and hundreds of batteries) always fail to start when they have to... Why is that ?

Survivorship bias. Press releases do not made when equipment work as expected

Re: OVH Incident in Strasbourg

#187

Post by switch007 »

nolok wrote: post #152
switch007 wrote: post #142

Earlier quoted context omitted.

> - naaah it's fine, we do not have time nor resources" Yup, been there multiple times in smaller hosting companies. It's basically how it goes. They don't get serious about outages until revenue is severely affected and the brand damaged, they don't get serious about security until there's been a big breach or sales are lost because of lack of certification.

Calling OVH a "smaller hosting company" (which you did do indirectly) is rather funny.

Yes, a comma would have made it slightly clear:

> Yup, been there multiple times, in smaller hosting companies.

I find it funny that anyone would think I would or could refer to them as small (and get away with it)

Re: OVH Incident in Strasbourg

#188

Post by spaghetti-guy »

PuffinBlue wrote: post #149
_wmd wrote: post #136

Earlier quoted context omitted.

While at $bigco we halted testing of generation equipment because it was sending DCs offline more often than it kept them up. Lawyers were involved, things got ugly

I'm completely unfamiliar with electrical generators/power generation, so take this question in the spirit of ignorance: Is there not a way to test generators without actually having them power the live datacenter infrastructure? I mean, simulate the exact generation and load requirements that the generators will face? I don't know if it's feasible to dump all that power to ground or whatever, but that way you could…

What you are talking about is 'load bank'. It's basically a massive hair dryer. Many data centres have these on the roof for exactly this purpose.

Re: OVH Incident in Strasbourg

#189

Post by abarringer »

gyaru wrote: post #105
wiz21c wrote: post #104

Damn, every emergency power supply I have encountered (the big ones with fuel and hundreds of batteries) always fail to start when they have to... Why is that ?

People not actually testing emergency equipment.

We had a generator that ran a two hour test every Thursday. It ran fine one Thursday, the next day we had a power outage and it failed to start because a capacitor went bad.

Re: OVH Incident in Strasbourg

#190

Post by sdfafhlska »

PuffinBlue wrote: post #149
_wmd wrote: post #136

Earlier quoted context omitted.

While at $bigco we halted testing of generation equipment because it was sending DCs offline more often than it kept them up. Lawyers were involved, things got ugly

I'm completely unfamiliar with electrical generators/power generation, so take this question in the spirit of ignorance: Is there not a way to test generators without actually having them power the live datacenter infrastructure? I mean, simulate the exact generation and load requirements that the generators will face? I don't know if it's feasible to dump all that power to ground or whatever, but that way you could…

You can if you have to. But then you're really only doing a fancy simulation.

I accompanied my dad (power engineer) to a water purification plant where they were testing new equipment for the back up generator. There their weekly tests involved moving the entire plant to the diesel generator and running it of back up power for a couple of hours (once you start a big generator you have to let it run or it wont last long).

Potential problems for your generator that a resistor bank wont capture include, power factor (phase shift from a motor or switching supply), harmonics (from switching power), startup transients (from every power supplies' capacitors).

All these things can trip the generator, or worse, burn it out.

So if you can't test with the real load, supersize it!

P.S. Every test is a simulation of reality. At Fukushima the diesel generators flooded. Lesson - the unknown reason that'll knock out your grid can knock out your backup

P.P.S If you can, gently turning the load back on is very beneficial. Don't flip the master switch that controls all your load - flip a part of your load, wait a while for the system to stabilize, and flip part of it back.

Post reply on HN