This is horrible and a sobering reminder to do the things we don't enjoy or consider -- disaster recovery.
How many of us here plan Fire Drills within our teams and larger organizations?
Fire declared in OVH SBG2 datacentre building
181–190 of 613 posts
Re: Fire declared in OVH SBG2 datacentre building
#182Puzzles and puzzle storm is down on Lichess: > Due to a fire at one of our data centres, a few of our servers are down and may be down permanently. We are restoring these servers from backups and will enable puzzles and storm as soon as possible. > > We hope that everyone who is dealing with the fire is safe, including the firefighters and everyone at OVH. <3.
I opened two tabs to relax tonight: Hacker News and Lichess. This was the top HN thread, and Lichess is having issues because of the fire. I didn't know what OVH was before 10 minutes ago, but this seems really impactful. I hope everyone there is safe and that the immediate disaster gets resolved quickly.
Look them up, they're one of the biggest hosting providers in the world ( especially in France), and due to cheap prices are especially pop-up with smaller scale stuff.
Re: Fire declared in OVH SBG2 datacentre building
#183I sent this story to my colleagues and one of them asked "where is the FM200?" I don't really know how FM200 systems work in data centres, but I'm guessing that if the fire didn't start from within the actual server room, FM200 might not save you? e.g. if a fire started elsewhere and went out of control, it would be able to burn through the walls/ceiling/floor of the server room, in which case no amount of FM200 gas…
I think most of these gases are or will eventually be banned in Europe because of their impact on the environment. I've seen newer datacenters use water mist sprays.
Re: Fire declared in OVH SBG2 datacentre building
#184Earlier quoted context omitted.
> I like knowing that I have a complete backup of my entire business within arm's reach. It could also provide a burglar a fantastic opportunity to pivot into career in data breaches.
This problem is usually solved through encryption.
If I were to ask my CISO if I was allowed to bring the production database home, I’m pretty sure his answer wouldn’t be “as long as you encrypt it”.
Re: Fire declared in OVH SBG2 datacentre building
#185The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…
Pictures from the fire: https://www.dna.fr/amp/faits-divers-justice/2021/03/10/stras...
is that entirely cargo containers? is that common for a data center?
Re: Fire declared in OVH SBG2 datacentre building
#186Earlier quoted context omitted.
OVH design their own datacenters, so it's possible that they missed something or some system or another didn't work as intended, thus the heavy damage.
They did not have a fire suppression system, only smoke detection. So yeah, they missed something.
Where did you get this?
Re: Fire declared in OVH SBG2 datacentre building
#187Earlier quoted context omitted.
Just guessing, but maybe a fire suppression system going off could wipe out all the machines? The couple datacenters I've been inside were small, old and used halon gas which wasn't supposed to destroy the machines. No idea how it works in big places these days.
A few years back there was an incident in Sweden where noise coming from the gas based fire suppression system going off destroyed hard drives [1]. 1. https://www.theregister.com/2018/04/26/decibels_destroy_disk...
ING in Romania had the same issue: https://www.datacenterdynamics.com/en/news/noise-from-fire-d...
Re: Fire declared in OVH SBG2 datacentre building
#188Earlier quoted context omitted.
They did not have a fire suppression system, only smoke detection. So yeah, they missed something.
Where did you get this?
Re: Fire declared in OVH SBG2 datacentre building
#189Earlier quoted context omitted.
This problem is usually solved through encryption.
If I were to ask my CISO if I was allowed to bring the production database home, I’m pretty sure his answer wouldn’t be “as long as you encrypt it”.
Once you're big enough to afford a CISO, you're likely big enough to afford office space with decent physical security to serve as a third replicated database site to complement your two datacenters.
These solutions are not one-size-fits-all. What works for a small startup isn't appropriate for a 100+ person company.
Re: Fire declared in OVH SBG2 datacentre building
#190Honestly just for the memes. https://isovhonfire.com
Probably not a good look for this to be the first thing another hosting company thinks to put up in response.
OVH aren't nice in that regard, and have trolled competitors to "leave it to the pros" before when there were serious incidents ( of which OVH have had their fair share), so it's not surprising.