Live data from Hacker News

OVH CEO Octave Klaba speaking about the incident [video]

ovh.com

101–110 of 143 posts

Re: OVH CEO Octave Klaba speaking about the incident [video]

#101

Are there industry options or methods of wiring to allow for a UPS room separate from the actual rooms the racks are stored in? It's almost tradition to have a rack with UPS's in the bottom and then the rest of the space filled with servers or drive arrays. We wouldn't ever think of putting a tiny backup generator in the bottom of every rack, so why do we put a battery storage system there? Also, with the advances in…

For the extreme opposite of that, Google famously trolled everyone 10 years ago by announcing that every one of their servers had its own in-chassis 12V battery: https://www.cnet.com/news/google-uncloaks-once-secret-server...

That was actually true, not an actual April Fool's day.

Re: OVH CEO Octave Klaba speaking about the incident [video]

#102

Are there industry options or methods of wiring to allow for a UPS room separate from the actual rooms the racks are stored in? It's almost tradition to have a rack with UPS's in the bottom and then the rest of the space filled with servers or drive arrays. We wouldn't ever think of putting a tiny backup generator in the bottom of every rack, so why do we put a battery storage system there? Also, with the advances in…

I've been led around a DC in the Netherlands where they used a separate room for the UPS batteries. Not only because of safety but because the reliability of small rack-mount UPSes really sucks (and you have to match them to the power draw of the servers in the rack or pay for a lot of unused capacity)

All racks and servers were connected to dual power feeds, so even when one of the feeds goes down the servers should run fine.

Re: OVH CEO Octave Klaba speaking about the incident [video]

#103
post #95
post #84

Earlier quoted context omitted.

> oxygen replacing suppressants No, they are a major risk to the employees working there. I would much rather have a data center destroyed by for every twenty years without victims than mandating the user of oxygen replacing fire suppressants.

My old company's DC uses FM200 gas which puts fires out without asphyxiating anyone (I'm assuming the experience would be a respiratory workout for anyone caught in it, but it would have cost £20000 to test).

Portable fire extinguishers:

- 16 USD (when buying in bulk 200+) [0]

- 236 USD ("Marine") [1]

12 Party balloons:

- 3 USD [2]

[0] https://mengfan.en.made-in-china.com/product/ksNEATWYXcVB/Ch...

[1] https://pacificfireandmarine.com/products/sea-fire-model-c-f...

[2] https://de.aliexpress.com/item/4000602253794.html

Re: OVH CEO Octave Klaba speaking about the incident [video]

#104
post #99

Lucky it wasn’t a UPS explosion. https://h2tools.org/lessons/battery-room-explosion

I realize it's probably just paranoia on my part but I am terrified of UPSs and can't bring myself to have one in my house. Gigantic batteries lying around in a flammable environment seems way too scary. My always-on server is read-only about 99% of the time, so I just put up with the outages when they happen (about 2 a year). If for some reason my OS eventually gets hosed because of this, I'll rebuild it. Maybe one…

A single lead-acid battery is putting out such a tiny amount of hydrogen. Put it in any room. It's a big fat nothing in a fire too: https://www.youtube.com/watch?v=ndfe0c00gwo Something could short but shorts are possible and dangerous even if no batteries are involved.

Or get a LiFePo battery and put it in a metal tub.

Re: OVH CEO Octave Klaba speaking about the incident [video]

#105
post #84

Earlier quoted context omitted.

If you're running an IT operation this big, you should have both fire detection units and oxygen replacing suppressants. That should be mandatory. Otherwise, it'd be very hard to contain. Especially when all your servers have RAID controllers with Li-Ion batteries or supercapacitors or other extremely trigger-happy components. Oh, and cooling systems. You're just kindling the fire with it at the beginning.

> oxygen replacing suppressants No, they are a major risk to the employees working there. I would much rather have a data center destroyed by for every twenty years without victims than mandating the user of oxygen replacing fire suppressants.

These days those systems are locked out when people are working in there.

Not so much in the old days :-)

I worked at palace (based at Cranfield uni) that had an experimental rig that used freon as a working fluid - we had emergency respirator sets (like the fire service uses) and people trained in them to use to rescue anyone.

Re: OVH CEO Octave Klaba speaking about the incident [video]

#106
post #99

Lucky it wasn’t a UPS explosion. https://h2tools.org/lessons/battery-room-explosion

I realize it's probably just paranoia on my part but I am terrified of UPSs and can't bring myself to have one in my house. Gigantic batteries lying around in a flammable environment seems way too scary. My always-on server is read-only about 99% of the time, so I just put up with the outages when they happen (about 2 a year). If for some reason my OS eventually gets hosed because of this, I'll rebuild it. Maybe one…

How do you feel about a Tesla in your Garage.

Re: OVH CEO Octave Klaba speaking about the incident [video]

#107
post #102

Are there industry options or methods of wiring to allow for a UPS room separate from the actual rooms the racks are stored in? It's almost tradition to have a rack with UPS's in the bottom and then the rest of the space filled with servers or drive arrays. We wouldn't ever think of putting a tiny backup generator in the bottom of every rack, so why do we put a battery storage system there? Also, with the advances in…

I've been led around a DC in the Netherlands where they used a separate room for the UPS batteries. Not only because of safety but because the reliability of small rack-mount UPSes really sucks (and you have to match them to the power draw of the servers in the rack or pay for a lot of unused capacity) All racks and servers were connected to dual power feeds, so even when one of the feeds goes down the servers should…

We do exactly the same you describe here in Latvia.

Re: OVH CEO Octave Klaba speaking about the incident [video]

#108

Are there industry options or methods of wiring to allow for a UPS room separate from the actual rooms the racks are stored in? It's almost tradition to have a rack with UPS's in the bottom and then the rest of the space filled with servers or drive arrays. We wouldn't ever think of putting a tiny backup generator in the bottom of every rack, so why do we put a battery storage system there? Also, with the advances in…

> "it's only a matter of time until Lithium chemistry batteries are available and that also increases the risk of fire." Not all Li-ion chemistries are equal. In particular, the increasingly popular LiFePO4 (LFP) technology has much higher energy density, longer lifespan, improved environmental characteristics, and similar if not better safety characteristics compared to lead-acid. (Besides sharing lead-acid's very l…

> lead-acid's very low risk of fire

Isn't lead-acid prone to releasing hydrogen gas?

When jump-starting a car, it is commonly recommended to connect the ground (black) cable to the chassis, not to the battery's black terminal, to avoid a spark igniting the hydrogen and causing an explosion.

Re: OVH CEO Octave Klaba speaking about the incident [video]

#109
post #67

Earlier quoted context omitted.

You would be using Halon injection in a normal DC, after sufficient warning to employees to gtfo if they hadn't done that already.

Not halon any more, but instead an ozone-friendly fire suppression gas such as argonite

That's basically true but I want to note that halon is a gas that actually suppresses fire while argonite is just there to get rid of oxygen.

Re: OVH CEO Octave Klaba speaking about the incident [video]

#110
post #103
post #95

Earlier quoted context omitted.

My old company's DC uses FM200 gas which puts fires out without asphyxiating anyone (I'm assuming the experience would be a respiratory workout for anyone caught in it, but it would have cost £20000 to test).

Portable fire extinguishers: - 16 USD (when buying in bulk 200+) [0] - 236 USD ("Marine") [1] 12 Party balloons: - 3 USD [2] [0] https://mengfan.en.made-in-china.com/product/ksNEATWYXcVB/Ch... [1] https://pacificfireandmarine.com/products/sea-fire-model-c-f... [2] https://de.aliexpress.com/item/4000602253794.html

Oxygen suppresion doesn't damage the equipment. Good luck cleaning out your equipment after you spray fire extinguisher all over them.
Post reply on HN