Interesting! I got the news from my local package courier website. They warn, their services can be unreliable due to the fire at OVH.
It's all connected.
Fire declared in OVH SBG2 datacentre building
251–260 of 613 posts
Re: Fire declared in OVH SBG2 datacentre building
#252My VPS in SBG3 stopped pinging around 9am.
My impression is that they tried very hard to maintain uptime, which was probably a bad idea when we see the extent of the damages. This VPS just hosts external facing services and is easy to set back up.
Re: Fire declared in OVH SBG2 datacentre building
#253Here is some pics of what it looked like before, spg 1-4 , and it’s history https://baxtel.com/data-center/ovh-strasbourg-campus
Is it just my laptop or is only ~40% of vertical screen real-estate dedicated to the actual content? :<
Nope, exactly the same here (on landscape phone).
Re: Fire declared in OVH SBG2 datacentre building
#254Earlier quoted context omitted.
There's a problem with testing sprinklers: engaging them can be damaging to contents and even structures. So, we're talking about completely emptying the facility, then taking it offline to dry for a time. I've never heard about this being done to anything that was already operational (but I wasn't researching this either).
Isn't something like Halon used in data centers for that reason? That can probably be tested without damaging infrastructure.
Halon is pretty much banned for years now, other agents have been introduced.
Sadly, making an actual full test of a gaseous extinguishing system (such as FM200 or Novec 1230) can be prohibitively expensive (mainly costs of the "reloading" of the system with new gas). Those are just mostly tested for correct pressure in tanks and if the detection electronics are working fine, making an actual dump would be very impractical (evacuation of personnel, ventilating the room afterwards etc.)
Re: Fire declared in OVH SBG2 datacentre building
#255Looks a little bit like Fukushima.
I hope the clean up doesn't take that long though..
Re: Fire declared in OVH SBG2 datacentre building
#256Earlier quoted context omitted.
> Hot spare on a different continent Just be cautious about data locality laws (not likely to affect you as joe average, more for businesses)
yeah, there's lots of countries with regulations that certain data can't leave the geographical boundary of the country. Often, it is the most sensitive data.
These laws generally don't work how people think they do.
For example, the Russian data residency law states that a copy of the data must be stored domestically, not that it can't be replicated outside the country.
The UAE has poorly written laws that have different regulations for different types of data - including fun stuff like only being subject to specific requirements if the data enters a 270 acre business park in Dubai.
Don't even get me started on storing encrypted data in one country and the keys in another...
Re: Fire declared in OVH SBG2 datacentre building
#257The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…
Here's what i do for my homelab setup that has a few machines running locally and some VPSes "in the cloud":
I personally have almost all of the software running in containers with an orchestrator on top (Docker Swarm in my case, others may also use Nomad, Kubernetes or something else). That way, rescheduling services on different nodes becomes less of a hassle in case of any one of them failing, since i know what should be running and what configuration i expect it to have, as well as what data needs to be persisted.
At the moment i'm using Time4VPS ( affiliate link: https://www.time4vps.com/?affid=5294 ) for the stuff that needs decent availability and because they're cheaper than almost all of the alternatives i've looked at (DigitalOcean, Vultr, Scaleway, AWS, Azure) and that matters to me.
Now, in case the entire data centre disappears, all of my data would still be available on a few HDDs under my desk (which are then replicated to other HDDs with rsync locally), given that i use BackupPC for incremental scheduled backups with rsync: https://backuppc.github.io/backuppc/
For simplicity, the containers also use bind mounts, so all of the data is readable directly from the file system, for example, under /docker (not really following some of the *nix file system layout practices, but this works for me because it's really easy to tell where the data that i want is).
I actually had to migrate over to a new node a while back, took around 30 minutes in total (updating DNS records included). Ansible can also really help with configuring new nodes. I'm not saying that my setup would work for most people or even anything past startups, but it seems sufficient for my homelab/VPS needs.
My conclusions:
- containers are pretty useful for reproducing software across servers
- knowing exactly which data you want to preserve (such as /var/lib/postgresql/data/pgdata) is also pretty useful, even though a lot of software doesn't really play nicely with the idea
- backups and incremental backups are pretty doable even without relying on a particular platform's offerings, BackupPC is more than competent and buying HDDs is far more cost effective than renting that space
- automatic failover (both DNS and moving the data to a new node) seems complicated, as does using distributed file systems; those are probably useful but far beyond what i actually want to spend time on in my homelab
- you should still check your backupsRe: Fire declared in OVH SBG2 datacentre building
#258Was the building made from stacked shipping containers? Containers are such a budget-friendly and trendy structural building block these days. They even click with the software engineers - "Hey, it's like Docker". Containers would seem to be at a disadvantage when it comes to dissipating, rather than containing, heat. I hope improved thermal management and fire suppression designs can be implemented.
I could be wrong, but I understood from other comments that only SBG1 used containers.
Re: Fire declared in OVH SBG2 datacentre building
#259I can't see anything about a fire suppression system mentioned? Doesn't OVH have one, except for colocation datacenters? A fire detection system using eg. lasers and Inergen(or Argonite) for putting the fire out is commonly used in datacenters. The gas fills the room and reduces the amount of oxygen in the room so most fires are put out within a minute. The cool thing is that the gas is designed to be used in rooms w…
So how is it safe with people if there is no oxygen left to breathe? Reminds me of my first trip to a datacenter, where the guy who accompanied us said: "In the event of a fire this room is filled with nitrogen in 20 seconds. But don't worry: nitrogen is not toxic!" Well, I was a little worried :)
Newer systems like the ones mentioned above are designed to reduce the amount of oxygen in the room to around 12% (down from around 21%). That's low enough to extinguish fires, but allows people to safely evacuate and prevent people from suffocating if they're incapacitated.
Re: Fire declared in OVH SBG2 datacentre building
#260Earlier quoted context omitted.
If I were to ask my CISO if I was allowed to bring the production database home, I’m pretty sure his answer wouldn’t be “as long as you encrypt it”.
That's because he doesn't trust you with this data. That has nothing to do with encryption safety. There is nothing magical about data centers making them safe while your local copy isn't.
> There is nothing magical about data centers making them safe while your local copy isn't.
Is this a serious comment? My house is not certified as being compliant with any security standards. Here's the list that the 3rd party datacenter we use is certified as complaint with:
https://aws.amazon.com/compliance/programs/
The data centers we operate ourselves are audited against several of those standards too. I guess you're right that there's nothing magic about security controls, but it has nothing to do with trust. Sensitive data should generally never leave a secure facility, outside of particularly controlled circumstances.