Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OVH Incident in Strasbourg

status.ovh.com

Post reply on HN
Print view View original

201–207 of 207 posts

Re: OVH Incident in Strasbourg

#201

Post by dredmorbius »

michaelt wrote: post #160
gyaru wrote: post #105

Earlier quoted context omitted.

People not actually testing emergency equipment.

I've seen places that /did/ test their backup power - but they got failures anyway, because of faults the test didn't reveal. For example you switch off the data centre circuit breakers and everything fails over to generators just fine. Test successful, right? Then when there's a real outage you have problems because the operations team's computers have all gone off, so they can't migrate load to a different data cen…

This is why chaos monkeys are a good thing. Regular drills too.

But you want to test the things you haven't thought about.

Re: OVH Incident in Strasbourg

#202

Post by dredmorbius »

merb wrote: post #200
sdfafhlska wrote: post #190

Earlier quoted context omitted.

You can if you have to. But then you're really only doing a fancy simulation. I accompanied my dad (power engineer) to a water purification plant where they were testing new equipment for the back up generator. There their weekly tests involved moving the entire plant to the diesel generator and running it of back up power for a couple of hours (once you start a big generator you have to let it run or it wont last lo…

P.S. Every test is a simulation of reality. At Fukushima the diesel generators flooded. Lesson - the unknown reason that'll knock out your grid can knock out your backup well the lesson there was more like, that it's stupid to put your diesel generators deep in the ground when they should sustain burst sea level raises. (well I think it's never a good idea to do that, I've seen special places to put them even deep in…

The generators were placed low to ensure they would not be disabled by a major earthquake, as shaking intensity increases with height above ground.

The basement was, according to plan, protected by a seawall.

The height of the seawall failed to take into account the fact that on a subduction tectonic plate, as pressure builds, the land-side plate rises, and as the earthquake relieving that pressure strikes, the land falls -- by as much as several meters.

The seawall's height failed to account for this.

That among other elements, but it proved sufficient to kick off the Fukushima disaster, given other aspects.

In hindsight, placing the generators at ground level in an elevated location might have been a better bet. Or locating the entire generating plant further upslope.

Re: OVH Incident in Strasbourg

#204

Post by dredmorbius »

IgorPartola wrote: post #176
waz0wski wrote: post #159

Earlier quoted context omitted.

Azamat suggest to follow for your Kazakh tech need @DNS_BORAT https://twitter.com/DNS_BORAT @InfoSecBorat https://twitter.com/InfoSecBorat @KanbanBorat https://twitter.com/KanbanBorat @mysqlborat https://twitter.com/mysqlborat @NetEng_Borat https://twitter.com/NetEng_Borat @secure_borat https://twitter.com/secure_borat @SecurityBorat https://twitter.com/SecurityBorat @Sysadm_Borat https://twitter.com/Sysadm_Borat

Anyone here remember https://en.m.wikipedia.org/wiki/Bastard_Operator_From_Hell

https://www.theregister.co.uk/data_centre/bofh/

Re: OVH Incident in Strasbourg

#205

Post by hvidgaard »

zaarn wrote: post #95
hvidgaard wrote: post #62

Earlier quoted context omitted.

If your generators are not reliable in the cold, the issue is the placement of them. This is something basic to account for.

The problem is usually not "not reliable in cold" but rather, the generators are X years old and the temperature is now changing in Europe from "mostly warm" to "warm over the day and icecold in the night" and finally aiming for "icecold all day", which means any equipment exposed will go through rather severe temperature changes. While generators are usually able to handle this with sufficiently low failure risk, th…

Europe sees that cycle every year, and have been for more than a century, so if they cannot handle that, they should be inside in a heated room.

Re: OVH Incident in Strasbourg

#207

Post by merb »

dredmorbius wrote: post #202
merb wrote: post #200

Earlier quoted context omitted.

P.S. Every test is a simulation of reality. At Fukushima the diesel generators flooded. Lesson - the unknown reason that'll knock out your grid can knock out your backup well the lesson there was more like, that it's stupid to put your diesel generators deep in the ground when they should sustain burst sea level raises. (well I think it's never a good idea to do that, I've seen special places to put them even deep in…

The generators were placed low to ensure they would not be disabled by a major earthquake, as shaking intensity increases with height above ground. The basement was, according to plan, protected by a seawall. The height of the seawall failed to take into account the fact that on a subduction tectonic plate, as pressure builds, the land-side plate rises, and as the earthquake relieving that pressure strikes, the land…

> In hindsight, placing the generators at ground level in an elevated location might have been a better bet. Or locating the entire generating plant further upslope.

yeah, well thats what I meant. I mean they were buried deep... There were even studies that this was dumb:

- https://news.usc.edu/86362/fukushima-disaster-was-preventabl...

- https://en.wikipedia.org/wiki/Fukushima_Daiichi_Nuclear_Powe... (section end)

- http://carnegieendowment.org/2012/03/06/why-fukushima-was-pr...

and basically tepco knew that. they were just too lazy (it was prolly too cost intensive) to do something against it (i.e. placing them higher or creating water bunkers (u-boot style)..

besides the generators there was also the human failure part. (most of the time human failure happens. I mean If I do not automate something, I might do it right 3 times and the fourth time I most often fail hard...)

Post reply on HN