61–70 of 143 posts
Earlier quoted context omitted.
a functioning fire suppression system would have probably been better but second to that i guess a mic would be a good investment
So putting water on servers don't destroy them?
Are there industry options or methods of wiring to allow for a UPS room separate from the actual rooms the racks are stored in? It's almost tradition to have a rack with UPS's in the bottom and then the rest of the space filled with servers or drive arrays. We wouldn't ever think of putting a tiny backup generator in the bottom of every rack, so why do we put a battery storage system there? Also, with the advances in…
If it turns out it's the upses that caught fire, one can't help but wonder if it would be a better idea to house the upses/backup power solutions in an adjacent smaller building outfitted with sprinklers perhaps?
The problem with water in the same place with high power equipment is that it instantly turns the room into a death trap for any personnel, now everything is potentially live. Also, in the first part of a lithium battery fire, dropping water on them is quite explosive. It will eventually quench the fire but on the short run it will make it worse, filling the room with explosive hydrogen and poisonous lithium hydroxid…
This would be one of inherient difference between smaller vs. giga players in cloud hosting. AWS/Google/Azure, if this happens, there should only be limited outage to a small fraction of customers. As a matter of fact, Google had such an incident before, and literally no customers (internal and external) noticed.
When you buy a storage API, sure, failure rates go up, latency increases 100x, but after a few hours its probably back to normal.
Of course, with the increased abstraction, you get more problems. "Availability zones" are useless when most cloud outages are because of configuration or systemic issues that tend to bring the whole thing down, no matter which AZ you are. But apparently it's now considered "good enough" to just go "oh we are down because AWS is down".
Not to be rude, but it's really wild to me that even after all this time during the pandemic, the CEO doesn't have a headset he can use so that the audio is intelligible. It's gotta be one of the highest ROI investments you can possibly make at this point.
a functioning fire suppression system would have probably been better but second to that i guess a mic would be a good investment
https://lafibre.info/ovh-datacenter/ovh-et-la-protection-inc... (posted in previous threads)
Earlier quoted context omitted.
The problem with water in the same place with high power equipment is that it instantly turns the room into a death trap for any personnel, now everything is potentially live. Also, in the first part of a lithium battery fire, dropping water on them is quite explosive. It will eventually quench the fire but on the short run it will make it worse, filling the room with explosive hydrogen and poisonous lithium hydroxid…
You would be using Halon injection in a normal DC, after sufficient warning to employees to gtfo if they hadn't done that already.
tl;dr UPS maintenance was performed by a vendor the day before the fire. Fire department used a thermal camera to isolate source of fire, it seemed to originate with 2 UPSes, one of which was the recently maintained UPS
Other random bits: SBG-2 was an older generation datacenter, had ventilation issues? They have 4 other datacenters who have a similar design. Others, including SBG-3, have newer designs. They're building 2500 servers per week. For the offline buildings that are not destroyed, they have to rebuild the electrical distribution and network. It was not clear if they are also moving servers physically.
If it turns out it's the upses that caught fire, one can't help but wonder if it would be a better idea to house the upses/backup power solutions in an adjacent smaller building outfitted with sprinklers perhaps?
The problem with water in the same place with high power equipment is that it instantly turns the room into a death trap for any personnel, now everything is potentially live. Also, in the first part of a lithium battery fire, dropping water on them is quite explosive. It will eventually quench the fire but on the short run it will make it worse, filling the room with explosive hydrogen and poisonous lithium hydroxid…
Most high-end UPSes have a relay where you can run an active-high or active-low emergency power off (EPO) signal. The EPO can either be a button that is pressed manually by the staff, automatically via fire suppression system, or both/either.
Schneider-APC white paper (PDF)
* https://download.schneider-electric.com/files?p_File_Name=AS...
The EPO can also cut-off the HVAC so oxygen is no longer fed into the area, and smoke isn't (re-)circulated.
In the US, this is probably covered in NFPA 75, "Standard for the Fire Protection of Information Technology Equipment":
* https://www.nfpa.org/codes-and-standards/all-codes-and-stand...
Earlier quoted context omitted.
You would be using Halon injection in a normal DC, after sufficient warning to employees to gtfo if they hadn't done that already.
Not halon any more, but instead an ozone-friendly fire suppression gas such as argonite