elliot..is it you?
Fire declared in OVH SBG2 datacentre building
151–160 of 613 posts
Re: Fire declared in OVH SBG2 datacentre building
#152The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…
We have two servers at OVH (RBX and GRA, not SBG). I make backups of all containers and VMs every day and keep the last three, plus one one each month. Backups are stored in a separate OVH storage disk and also downloaded to a NAS on-premise. In case of a disaster, we'd have to rent a new server, reprovision the VMs and containers and restore the backups. About two days of work to make sure everything works fine and we could lose about 24 hours of data.
It's not the best in terms of Disaster Recovery Plan but we accept that level of risk.
Re: Fire declared in OVH SBG2 datacentre building
#153Unfortunately A lot of people are going to find out the hard way today why AWS/GCP/Big Expensive Cloud is so expensive (Hint: they have redundancy and failover procedures which drive up costs). Keep in mind I’m talking not of “downtime” but of actual data loss which might affect business continuity. This is really tragic. I’m hoping they have some kind of multi regional backup/replication and not just multi zones (al…
Yup, I've never heard of a fire taking out a Big Cloud DC. They actually know what they're doing and don't put server racks in shipping containers stacked on top of each other. If you want quality in life, sometimes you have to pay for it.
Personally I'll continue to use these third world cloud providers. But I like to live on the edge.
Re: Fire declared in OVH SBG2 datacentre building
#154Was the building made from stacked shipping containers? Containers are such a budget-friendly and trendy structural building block these days. They even click with the software engineers - "Hey, it's like Docker". Containers would seem to be at a disadvantage when it comes to dissipating, rather than containing, heat. I hope improved thermal management and fire suppression designs can be implemented.
OVH use a custom water cooling solution they claim enables these niche designs and increases rack density.
As another comment says, that density probably exacerbated the issue here.
Re: Fire declared in OVH SBG2 datacentre building
#155The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…
I'm at OVH as well (in the BHS datacenter, fortunately). I run my entire production system on one beefy machine. The apps and database are replicated to a backup machine hosted with Hetzner (in their Germany datacenter). I also run a tiny VM at OVH which proxies all traffic to Hetzner. I use a failover IP to point at the big rig at OVH. If the main machine fails, I move the failover IP to the VM, which sends all traf…
Are you me, by any chance? :-)
I also run our entire production system on one beefy machine at OVH, and replicate to a similar machine at Hetzner. In case of a failure, we just change DNS, which has a 1 hour TTL. We've needed to do an unplanned fail-over only once in over 10 years.
And like you, I have an extra replica at the office, because it feels safe having a physical copy of the data literally at hand.
Re: Fire declared in OVH SBG2 datacentre building
#156Re: Fire declared in OVH SBG2 datacentre building
#157The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…
Pictures from the fire: https://www.dna.fr/amp/faits-divers-justice/2021/03/10/stras...
Re: Fire declared in OVH SBG2 datacentre building
#158I can't see anything about a fire suppression system mentioned? Doesn't OVH have one, except for colocation datacenters? A fire detection system using eg. lasers and Inergen(or Argonite) for putting the fire out is commonly used in datacenters. The gas fills the room and reduces the amount of oxygen in the room so most fires are put out within a minute. The cool thing is that the gas is designed to be used in rooms w…
What's the point of a fire suppression system that destroys what it should protect?
In a data center the fire suppression is mostly there to protect the servers.
Re: Fire declared in OVH SBG2 datacentre building
#159I sent this story to my colleagues and one of them asked "where is the FM200?"
I don't really know how FM200 systems work in data centres, but I'm guessing that if the fire didn't start from within the actual server room, FM200 might not save you? e.g. if a fire started elsewhere and went out of control, it would be able to burn through the walls/ceiling/floor of the server room, in which case no amount of FM200 gas can save you, right?
Another possibility, of course, is that the FM200 system simply failed to trigger even though the fire started from within the server room.
There is no published investigation details about this incident yet, I believe. Can somebody chime in about past incidents where FM200 failed to save the day?
Re: Fire declared in OVH SBG2 datacentre building
#160There is no cloud, there is just other people’s computers. And they’re on fire.
Your data was in the cloud, now it's in the smoke.