Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OVH CEO Octave Klaba speaking about the incident [video]

ovh.com

Post reply on HN
Print view View original

121–130 of 143 posts

Re: OVH CEO Octave Klaba speaking about the incident [video]

#121

Post by chrisandchris »

raphaelj wrote: post #77
tpmx wrote: post #24

I think the key issue here is that there wasn't a functioning fire suppresssion system in place. Second question: is such a system required for this kind of operation? Maybe?

I’m pretty confident they have efficient fire suppression systems. They are hosting at least 400,000 servers. They for sure have multiple servers taking fire every single day, and yet it’s the first time it ends up in a catastrophic fire. The fire suppression system either catastrophically failed, or something out of design happened with one of their inverters, as suggested in the video.

According to various sources (here on HN, also their official site) they only have fire detection and not suppresion [1].

[1] https://www.ovh.com/world/us/about-us/datacenters.xml

Re: OVH CEO Octave Klaba speaking about the incident [video]

#122

Post by Symbiote »

pcl wrote: post #89
bayindirh wrote: post #85

Earlier quoted context omitted.

I understand the concerns you have, however I think there's a good middle ground. Would you consider the following procedure acceptable? - Isolate all rooms with fire-proof doors. - Keep fire supression system at manual. - When fire breaks try to contain (we have 24h watch). - If fails trigger fire supression system. It has 90 second delay and activated per room. - Leave premsises, make the calls. Fire control and su…

Other industries use lock-out keys when people are in harm's way. It seems like it'd be easy enough to design an oxygen-replacement system that has lock-outs that people engage whenever they need to enter the protected rooms. https://en.wikipedia.org/wiki/Lockout–tagout

I think it's a bit more complicated, since the extinguishing system is intended to be used in emergencies.

The usual system, e.g. in a train maintenance depot:

- Employee is going to work underneath the train, which therefore must not be moved. He isolates the power supply, and locks this isolation with his padlock.

A situation in which it's suddenly very important to reconnect the power is extremely unlikely. If the employee forgets to remove his padlock, it's disruptive but not dangerous. (I've seen this system once, and when people left for the day they were supposed to lock their padlocks on a special board as part of clocking out.)

For the datacentre, if the fire alarm goes off, everyone is supposed to leave by the nearest exit -- not go back the way they came in, unlocking their padlocks to allow the extinguishing system to be used.

Re: OVH CEO Octave Klaba speaking about the incident [video]

#123

Post by rootusrootus »

Nextgrid wrote: post #115
branko_d wrote: post #108

Earlier quoted context omitted.

> lead-acid's very low risk of fire Isn't lead-acid prone to releasing hydrogen gas? When jump-starting a car, it is commonly recommended to connect the ground (black) cable to the chassis, not to the battery's black terminal, to avoid a spark igniting the hydrogen and causing an explosion.

Isn't hydrogen produced only while the battery is charging, or at worst, when discharging? If so, this still makes them much safer as they're otherwise inert while not in operation, compared to lithium-polymer which contains materials flammable at all times.

LiPo and LFP are very different things. LiPo is probably the biggest fire risk battery chemistry in common use today. LFP is pretty safe. LTO even more so.

Re: OVH CEO Octave Klaba speaking about the incident [video]

#124

Post by AYBABTME »

Mauricebranagh wrote: post #106
bscphil wrote: post #99

Earlier quoted context omitted.

I realize it's probably just paranoia on my part but I am terrified of UPSs and can't bring myself to have one in my house. Gigantic batteries lying around in a flammable environment seems way too scary. My always-on server is read-only about 99% of the time, so I just put up with the outages when they happen (about 2 a year). If for some reason my OS eventually gets hosed because of this, I'll rebuild it. Maybe one…

How do you feel about a Tesla in your Garage.

Or a 40L tank full of gasoline in your garage?

Re: OVH CEO Octave Klaba speaking about the incident [video]

#125

Post by doggodaddo78 »

deftnerd wrote: post #23

Are there industry options or methods of wiring to allow for a UPS room separate from the actual rooms the racks are stored in? It's almost tradition to have a rack with UPS's in the bottom and then the rest of the space filled with servers or drive arrays. We wouldn't ever think of putting a tiny backup generator in the bottom of every rack, so why do we put a battery storage system there? Also, with the advances in…

Batteries burn up, especially NMC Li ion.

Battery rooms were traditionally separate, used lead acid batteries, were surrounded by thicker walls, and equipped with FM200, just like main datacenter floors. They were typically placed near the transfer and PDU switchgear. I wouldn't put anything more flammable than LiFePO4 in a battery room, much less anywhere near a server.

It's people who decide to throw away conventions, common sense, and building codes because "they know better" who get into trouble.

I suspect this datacenter company could be sued into oblivion.

Re: OVH CEO Octave Klaba speaking about the incident [video]

#126

Post by doggodaddo78 »

tempay wrote: post #114
branko_d wrote: post #108

Earlier quoted context omitted.

> lead-acid's very low risk of fire Isn't lead-acid prone to releasing hydrogen gas? When jump-starting a car, it is commonly recommended to connect the ground (black) cable to the chassis, not to the battery's black terminal, to avoid a spark igniting the hydrogen and causing an explosion.

The lead-acid batteries used in data centres are normally "sealed lead-acid batteries" and release significantly less hydrogen. See: https://en.wikipedia.org/wiki/VRLA_battery

Wikipedia is wrong. VRLAs aren't SLAs, and vice-versa. Actual SLAs are very, very expensive. All VRLAs can vent, by definition.

VRLAs can overcharge and catch fire just like flooded batteries. They just don't go kaboom as much or spray acid everywhere. When having thousands of batteries in a room, one of them catching fire is inevitable. This is why batteries are in separate fire containment rooms, on nonflammable shelving, in redundant strings, and protected by FM200.

Re: OVH CEO Octave Klaba speaking about the incident [video]

#127

Post by lclarkmichalek »

LinuxBender wrote: post #73
mikepurvis wrote: post #34

Earlier quoted context omitted.

For the extreme opposite of that, Google famously trolled everyone 10 years ago by announcing that every one of their servers had its own in-chassis 12V battery: https://www.cnet.com/news/google-uncloaks-once-secret-server...

And Facebook had small UPS/ATS units at the end of each row. Not sure if they still do that today it was like that when I walked through their datacenter. They did that for the purpose of power efficiency. They lost far less power by having many smaller units.

What you saw likely wasn't a UPS, but an RPP

> A data center typically spans four rooms, called suites, where racks of servers are arranged in rows. Up to four MSBs provide power to each suite. In turn, each MSB supplies up to four 1.25 MW Switch Boards (SBs). From each SB, power is fed to the 190 KW Reactive Power Panels (RPPs) stationed at the end of each row of racks.

https://research.fb.com/wp-content/uploads/2016/11/dynamo_fa...

The UPS role is taken more by BBUs (Battery Backup Units). https://www.youtube.com/watch?v=KNsposM0sJE has lots of info about them.

(I work for FB, on entirely unrelated things)

Re: OVH CEO Octave Klaba speaking about the incident [video]

#128

Post by teh_klev »

lovedswain wrote: post #12

tl;dr UPS maintenance was performed by a vendor the day before the fire. Fire department used a thermal camera to isolate source of fire, it seemed to originate with 2 UPSes, one of which was the recently maintained UPS

It's always the bloody UPS after "maintenance" :)

Back when I worked in hosting we'd get an email from this particular DC's NOC about either "UPS Maintenance" or generator testing. Our hearts would sink because, during one particular eighteen month period, there was a 50/50 chance our suite would go dark afterwards.

Re: OVH CEO Octave Klaba speaking about the incident [video]

#129

Post by Macha »

Dylan16807 wrote: post #104
bscphil wrote: post #99

Earlier quoted context omitted.

I realize it's probably just paranoia on my part but I am terrified of UPSs and can't bring myself to have one in my house. Gigantic batteries lying around in a flammable environment seems way too scary. My always-on server is read-only about 99% of the time, so I just put up with the outages when they happen (about 2 a year). If for some reason my OS eventually gets hosed because of this, I'll rebuild it. Maybe one…

A single lead-acid battery is putting out such a tiny amount of hydrogen. Put it in any room. It's a big fat nothing in a fire too: https://www.youtube.com/watch?v=ndfe0c00gwo Something could short but shorts are possible and dangerous even if no batteries are involved. Or get a LiFePo battery and put it in a metal tub.

That looks like enough fire that spreading to my wooden floors or desks, carpets or skirting boards seems more likely than not?

Re: OVH CEO Octave Klaba speaking about the incident [video]

#130

Post by Macha »

AYBABTME wrote: post #124
Mauricebranagh wrote: post #106

Earlier quoted context omitted.

How do you feel about a Tesla in your Garage.

Or a 40L tank full of gasoline in your garage?

Fine, given it's outside, on the street, a couple of meters from the front of my house, as I live somewhere where garages are a luxury.
Post reply on HN