I hereby declare this to be a fire.
Fire declared in OVH SBG2 datacentre building
331–340 of 613 posts
Re: Fire declared in OVH SBG2 datacentre building
#332The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…
I only have a personal server running in Hetzner but it's mirrored onto a tiny local computer at home. They both run postfix + dovecot, so mail is synced via dovecot replication. Data is rsync-ed daily, and everything has ZFS snapshots. MySQL is not set into replication - my home internet breaks often enough to have serious issues, so instead I drop everything every day import a full dump from the main server, and do…
Not saying that you should never do a full mysql dump. Nor that you should not ensure that you can import a full dump.
But when you already use ZFS you can do a very speedy full backup with:
mysql Re: Fire declared in OVH SBG2 datacentre building
#333According to the official status page, the whole datacenter is still green http://status.ovh.com/vms/index_sbg2.html
Also, 3 years ago they had an outage in Strasbourg, and the status page was down apparently as a result of the outage. https://news.ycombinator.com/item?id=15661218 They are not the only ones though. All too common. Well, it's tricky to set this up properly. The only proper way would be to use external infra for the status page.
Isn’t that why things like statuspage.io exist, though?
Re: Fire declared in OVH SBG2 datacentre building
#334Earlier quoted context omitted.
They did not have a fire suppression system, only smoke detection. So yeah, they missed something.
There are photos of the fire suppression system in one of their data centres in this (French) forum thread: https://lafibre.info/ovh-datacenter/ovh-et-la-protection-inc... . They have sprinklers, with the reasoning being that the burning racks' data is gone anyway if there's a fire, and at least sprinklers don't accidentally kill the technicians.
Sprinklers are only present in OVH Canada datacenter ! There are no sprinklers in Europe ones.
Re: Fire declared in OVH SBG2 datacentre building
#335Earlier quoted context omitted.
Presumably you would have trouble breathing as it displaced the oxygen in the room?
Yeah, that was always my understanding. GP saying "the gas is designed to be used in rooms with people, so that [it] can be triggered any time" made me second-guess that though. Maybe there is a concentration of oxygen that is high enough for humans to survive, yet too low to sustain combustion?
> Maybe there is a concentration of oxygen that is high enough for humans to survive, yet too low to sustain combustion?
Yes, supposedly (12% ish?). Can't say I'd be thrilled at the idea of testing it.
Re: Fire declared in OVH SBG2 datacentre building
#336And this is why the big 3 will continue to dominate. AWS, Microsoft and Google can throw in a lot more money at their phyiscal infrastructure than any other cloud provider. After this sorry episode, I dont think any CTO or CIO of any public company will be able to even consider using the other guys. edit: I am not implying that we put all eggs in one basket with no failover and dr. I am implying the big cos will pay…
I could replicate my whole infrastructure on 3 different OVH datacenters, with enough provision to support twice the peak load - it would still be cheaper than a single infrastructure at AWS, and I would get a better uptime than AWS: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
I agree with you 100%. However, I did state * any CTO or CIO of any public company* ... the executives don't worry about costs , they worry about being able to * project reliability * .
Re: Fire declared in OVH SBG2 datacentre building
#337Did OVH claim that SBG1 and SBG2 were isolated failure domains? Despite them just being different rooms in the same building?
They're separate buildings, with separate power systems, just standing next to another. Next to SBG2 are also SBG3 and SBG4.
Those buildings weren't quite as far apart as they should have been if a fire in one requires all 4 of the others to be turned off...
Re: Fire declared in OVH SBG2 datacentre building
#338Earlier quoted context omitted.
I think most of these gases are or will eventually be banned in Europe because of their impact on the environment. I've seen newer datacenters use water mist sprays.
You might be thinking of Halons, which are CFCs that depletes the ozone layer? They are mostly phased out worldwide but existing installations might still be in use. FM200 is something else that is often used in modern builds (not just datacenters).
It seems that HFC are being phased out too: https://ec.europa.eu/clima/policies/f-gas/legislation_en
Re: Fire declared in OVH SBG2 datacentre building
#339Earlier quoted context omitted.
For some context: “SBG1, the first Strasbourg data center, consisting of twelve containers, came online in 2012. The 12 containers had a capacity of 12,000 servers. SBG2 is a non-container data center in 2016 using its “Tower” design with a capacity of 30,000 servers. SBG3 tower was built in 2017 with a capacity of 30,000 servers. SBG4 was built in 2013 as several containers to augment capacity, but was decommissione…
and a 2017 (!) article where they were planning to remove 1 and 2 because of power issues! > OVH to Disassemble Container Data Centers after Epic Outage in Europe > “This is probably the worst-case scenario that could have happened to us.” > OVH [...] is planning to shut down and disassemble two of the three data centers on its campus in Strasbourg, France, following a power outage that brought down the entire campus…
It sounds like the plan was to shutdown 1 and 4, the latter of which happened and the former which did not.
Re: Fire declared in OVH SBG2 datacentre building
#340Earlier quoted context omitted.
I could replicate my whole infrastructure on 3 different OVH datacenters, with enough provision to support twice the peak load - it would still be cheaper than a single infrastructure at AWS, and I would get a better uptime than AWS: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
I agree with you 100%. However, I did state * any CTO or CIO of any public company* ... the executives don't worry about costs , they worry about being able to * project reliability * .
Executives that worry about reliability would insist on deploying on multiple data centers, which would make the project more reliable than any single AWS availability zone.
Also, cost matters if the AWS bill is one of the company's top expenses.