Earlier quoted context omitted.
a functioning fire suppression system would have probably been better but second to that i guess a mic would be a good investment
So putting water on servers don't destroy them?
[deleted]
OVH CEO Octave Klaba speaking about the incident [video]
51–60 of 143 posts
Re: OVH CEO Octave Klaba speaking about the incident [video]
#52I think the key issue here is that there wasn't a functioning fire suppresssion system in place. Second question: is such a system required for this kind of operation? Maybe?
If you're running an IT operation this big, you should have both fire detection units and oxygen replacing suppressants.
That should be mandatory. Otherwise, it'd be very hard to contain. Especially when all your servers have RAID controllers with Li-Ion batteries or supercapacitors or other extremely trigger-happy components.
Oh, and cooling systems. You're just kindling the fire with it at the beginning.
Re: OVH CEO Octave Klaba speaking about the incident [video]
#53Earlier quoted context omitted.
a functioning fire suppression system would have probably been better but second to that i guess a mic would be a good investment
So putting water on servers don't destroy them?
This is typically a FM200 system in the datacenter [1] which is a pressurized gas. There are also often water "dry-pipes" in place that can pressurize if the FM200 fails. Fire suppression codes vary by country.
[1] - https://en.wikipedia.org/wiki/Automatic_fire_suppression
Re: OVH CEO Octave Klaba speaking about the incident [video]
#54Earlier quoted context omitted.
For the extreme opposite of that, Google famously trolled everyone 10 years ago by announcing that every one of their servers had its own in-chassis 12V battery: https://www.cnet.com/news/google-uncloaks-once-secret-server...
Yes but Google also later moved to a 48VDC power architecture with the batteries at the bottom of the rack. See http://apec.dev.itswebs.com/Portals/0/APEC%202017%20Files/Pl... page 6.
That presentation is pretty interesting, including Google inventing their own "Switched Tank" DC-DC converters because the existing ones weren't efficient or reliable enough.
Re: OVH CEO Octave Klaba speaking about the incident [video]
#55Are there industry options or methods of wiring to allow for a UPS room separate from the actual rooms the racks are stored in? It's almost tradition to have a rack with UPS's in the bottom and then the rest of the space filled with servers or drive arrays. We wouldn't ever think of putting a tiny backup generator in the bottom of every rack, so why do we put a battery storage system there? Also, with the advances in…
Depends for a DC / Telco your normally feeding 48v DC from the UPS which is normally separate.
The lack of fire suppression is also very worrying.
Re: OVH CEO Octave Klaba speaking about the incident [video]
#56Earlier quoted context omitted.
a functioning fire suppression system would have probably been better but second to that i guess a mic would be a good investment
So putting water on servers don't destroy them?
There are gas based fire suppression system such as FM-200 and Novec 1230.
If you aren't using gas based fire suppression systems it is basically an amateur operation. The small data centre at my work (10-ish racks) has a FM-200 fire suppression system, it really isn't that expensive to set up.
Re: OVH CEO Octave Klaba speaking about the incident [video]
#57Earlier quoted context omitted.
Yes but Google also later moved to a 48VDC power architecture with the batteries at the bottom of the rack. See http://apec.dev.itswebs.com/Portals/0/APEC%202017%20Files/Pl... page 6.
That presentation is pretty interesting, including Google inventing their own "Switched Tank" DC-DC converters because the existing ones weren't efficient or reliable enough.
There's a whole separate paper on that:
https://storage.googleapis.com/pub-tools-public-publication-...
Re: OVH CEO Octave Klaba speaking about the incident [video]
#58Earlier quoted context omitted.
On the other hand, just about every month, there's a story on HN saying why are you wasting your money on AWS when OVH is so much cheaper (for example https://news.ycombinator.com/item?id=24966028 ). And well, I guess this is one of the reasons.
If you choose to run 100% of your workload on a single EC2 VM in us-east-chaos-monkey and put nothing in S3, only local mounted block storage that also disappears when you reboot your on-demand EC2, that is on you.
2 OVH dedicated servers in different countries are still cheaper than one AWS instance.
e.g. I've got servers at OVH SBG-2, Hetzner's Falckenstein, and Online.net's AMS datacenters — the total of which is still almost a magnitude less than the same cost on AWS or GCP (granted, that's including traffic)
Re: OVH CEO Octave Klaba speaking about the incident [video]
#59Earlier quoted context omitted.
If you choose to run 100% of your workload on a single EC2 VM in us-east-chaos-monkey and put nothing in S3, only local mounted block storage that also disappears when you reboot your on-demand EC2, that is on you.
2 OVH dedicated servers in different countries are still cheaper than one AWS instance. e.g. I've got servers at OVH SBG-2, Hetzner's Falckenstein, and Online.net's AMS datacenters — the total of which is still almost a magnitude less than the same cost on AWS or GCP (granted, that's including traffic)
Pretty much. Use 5% of the money you saved moving your workload from AWS to OVH to support failover, DR, and backups. You can probably buy like five or six machines of equivalent spec or more.
Re: OVH CEO Octave Klaba speaking about the incident [video]
#60If it turns out it's the upses that caught fire, one can't help but wonder if it would be a better idea to house the upses/backup power solutions in an adjacent smaller building outfitted with sprinklers perhaps?
The problem with water in the same place with high power equipment is that it instantly turns the room into a death trap for any personnel, now everything is potentially live. Also, in the first part of a lithium battery fire, dropping water on them is quite explosive. It will eventually quench the fire but on the short run it will make it worse, filling the room with explosive hydrogen and poisonous lithium hydroxid…
You would not be using water here