Did OVH claim that SBG1 and SBG2 were isolated failure domains? Despite them just being different rooms in the same building?
Fire declared in OVH SBG2 datacentre building
301–310 of 613 posts
Re: Fire declared in OVH SBG2 datacentre building
#302Earlier quoted context omitted.
Are you truly paranoid? If my money and/or job depended on having something running without (or with minimal) disruption I would be as paranoid as you, too. BTW - Some people call this business recovery plan, not plain paranoia ;-)
Enterprise-level projects often have only light protection against wrongful hosting account termination, reasoning that spending a lot of money and having an account manager keeps them safe from clumsy automated systems. So they might have their primary and replica databases at different DCs from the same hosting provider, and only their nightly backup to a different provider. Four copies to four different providers…
This is why you should have a “Cloud 3-2-1” backup plan. Have 3 copies of your data, two with your primary provider, and 1 with another.
e.g., if you are an AWS customer, have your back ups in S3 and use simple replication to sync that to either GCS or Azure, where you can get the same level of compliance attestation as from AWS.
Re: Fire declared in OVH SBG2 datacentre building
#303I sent this story to my colleagues and one of them asked "where is the FM200?" I don't really know how FM200 systems work in data centres, but I'm guessing that if the fire didn't start from within the actual server room, FM200 might not save you? e.g. if a fire started elsewhere and went out of control, it would be able to burn through the walls/ceiling/floor of the server room, in which case no amount of FM200 gas…
I think most of these gases are or will eventually be banned in Europe because of their impact on the environment. I've seen newer datacenters use water mist sprays.
You might be thinking of Halons, which are CFCs that depletes the ozone layer? They are mostly phased out worldwide but existing installations might still be in use.
FM200 is something else that is often used in modern builds (not just datacenters).
Re: Fire declared in OVH SBG2 datacentre building
#304I can't see anything about a fire suppression system mentioned? Doesn't OVH have one, except for colocation datacenters? A fire detection system using eg. lasers and Inergen(or Argonite) for putting the fire out is commonly used in datacenters. The gas fills the room and reduces the amount of oxygen in the room so most fires are put out within a minute. The cool thing is that the gas is designed to be used in rooms w…
Are fires common in data centres? Specialized fire suppression tech seems to indicate that they are.
Yes, mostly due to two reasons:
* overall high energy density (lots of current flowing everywhere)
* the batteries for backup power are dangerous, and can easily(ish) overheat when activated.
Re: Fire declared in OVH SBG2 datacentre building
#305According to the official status page, the whole datacenter is still green http://status.ovh.com/vms/index_sbg2.html
Also, 3 years ago they had an outage in Strasbourg, and the status page was down apparently as a result of the outage.
https://news.ycombinator.com/item?id=15661218
They are not the only ones though. All too common. Well, it's tricky to set this up properly. The only proper way would be to use external infra for the status page.
Re: Fire declared in OVH SBG2 datacentre building
#306Earlier quoted context omitted.
You're completely right. IMO this is the best comment in this whole thread. Their status page must be broken, or it's a lie.
If there is a SLA with consequences associated with it every status page is going to be a lie.
Well, it sucks to catch fire and I care for the employees and the firemen, but if their status page is a lie then I have a whole lot less sympathy for the business. That's shady business and they should feel bad.
I can appreciate an honest mistake though, like the status page server cron is hosted in the same cluster that caught fire and hence it burnt down and can't update the page anymore.
Re: Fire declared in OVH SBG2 datacentre building
#307Earlier quoted context omitted.
In my case: * All my services are dockerized and have gitlab pipelines to deploy on a kubernetes cluster (RKE/K3s/baremetal-k8s) * git repo's containing the build scripts/pipelines are replicated on my gitlab instance and multiple work computers (laptop & desktop) * Data and databases are regularly dumped and stored in S3 and my home server * Most of the infrastructure setup (AWS/DO/Azure, installing kubernetes) is i…
How often do you test starting from a clean slate?
I have a similar setup: I recreate everything for every major kubernetes update.
Re: Fire declared in OVH SBG2 datacentre building
#308Earlier quoted context omitted.
That's because he doesn't trust you with this data. That has nothing to do with encryption safety. There is nothing magical about data centers making them safe while your local copy isn't.
> There is nothing magical about data centers making them safe while your local copy isn't. Is this a serious comment? My house is not certified as being compliant with any security standards. Here's the list that the 3rd party datacenter we use is certified as complaint with: https://aws.amazon.com/compliance/programs/ The data centers we operate ourselves are audited against several of those standards too. I guess…
Of course it's serious.
You are entierly missing the point by quoting the compliance programs followed by AWS whose sole business is being a third party hoster.
For most business, what you call sensitive data is customers and orders listing, payment history, inventory if you are dealing in physical goods and HR related files. These are not state secrets. Encryption and a modicum of physical security go a long way.
I personally find the idea that you shouldn't store a local backup of this kind of data out of security concern entirely laughable. But that's me.
Re: Fire declared in OVH SBG2 datacentre building
#309I just recently started moving some services for my business to one of OVH's US-based data centers. Should I take this fire as evidence that OVH is incompetent and get out? I really don't want AWS, or the big three hyperscalers in general, to be the only option.
Re: Fire declared in OVH SBG2 datacentre building
#310Did OVH claim that SBG1 and SBG2 were isolated failure domains? Despite them just being different rooms in the same building?
They're separate buildings, with separate power systems, just standing next to another. Next to SBG2 are also SBG3 and SBG4.