Live data from Hacker News

Fire declared in OVH SBG2 datacentre building

travaux.ovh.net

291–300 of 613 posts

Re: Fire declared in OVH SBG2 datacentre building

#291
post #263
post #214

Earlier quoted context omitted.

Easy to see now that a lightly constructed five story cube might not be a fully fire proof.

Would you humor us with a link to a fully fire proof datacenter?

I think Microsoft had some experimental datacenter containers they submerged in the northern Atlantic for passive cooling, and I believe those were filled with an inert gas as well. I guess that would come very close to an actual fireproof datacenter.

Re: Fire declared in OVH SBG2 datacentre building

#292
post #170
post #133

Earlier quoted context omitted.

Or disaster prevention. I'm curious how their fire suppression system failed so spectacularly..

I think they failed to install it. They only mention smoke detection: https://www.ovh.com/world/us/about-us/datacenters.xml

In French but with pictures: https://lafibre.info/ovh-datacenter/ovh-et-la-protection-inc... - probably a different OVH data centre, but they clearly have sprinklers there. The argument they made against gas extinguishers is by the time they have to use it, the data is gone anyway, and it's only going to trigger in the affected areas. It's also far safer for the people working there.

Re: Fire declared in OVH SBG2 datacentre building

#293

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…

Also stupid things not to forget: make sure your dns provider is independent otherwise you won’t be able to point to your new server (or have a secondary DNS provider). Make sure any email required for 2FA or communicating with your hosting service managing your infrastructure isn’t running on that same infrastructure.

Re: Fire declared in OVH SBG2 datacentre building

#294

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…

I only have a personal server running in Hetzner but it's mirrored onto a tiny local computer at home.

They both run postfix + dovecot, so mail is synced via dovecot replication. Data is rsync-ed daily, and everything has ZFS snapshots. MySQL is not set into replication - my home internet breaks often enough to have serious issues, so instead I drop everything every day import a full dump from the main server, and do a local dump as backup on both sides.

I don't have automatic failover set up.

Re: Fire declared in OVH SBG2 datacentre building

#295
post #146

According to the official status page, the whole datacenter is still green http://status.ovh.com/vms/index_sbg2.html

You're completely right. IMO this is the best comment in this whole thread. Their status page must be broken, or it's a lie.

If there is a SLA with consequences associated with it every status page is going to be a lie.

Re: Fire declared in OVH SBG2 datacentre building

#296
post #180
post #175

Earlier quoted context omitted.

OVH design their own datacenters, so it's possible that they missed something or some system or another didn't work as intended, thus the heavy damage.

They did not have a fire suppression system, only smoke detection. So yeah, they missed something.

There are photos of the fire suppression system in one of their data centres in this (French) forum thread: https://lafibre.info/ovh-datacenter/ovh-et-la-protection-inc.... They have sprinklers, with the reasoning being that the burning racks' data is gone anyway if there's a fire, and at least sprinklers don't accidentally kill the technicians.

Re: Fire declared in OVH SBG2 datacentre building

#298
post #55

Earlier quoted context omitted.

I'm not sure if it's because my tolerance of Graham Linehan has snapped or not, but I barely laugh at the IT Crowd any more. As with other GL shows I find it's just mostly held together but the cast's delivery and such The laugh track and the writing is honestly dated even by the standards of Dads Army.

I don't remember the details, but I think that season 2 kind of retroactively ruined season 1. They used to have all those O'Reilly and EFF stickers, and working at a help desk at the time, it felt very authentic. Then everything got super nice in season 2 -- leather couches, people were dressing nicely, etc. It kind of lost its charm. You can't rewatch it because you know Denholm is just going to randomly jump out o…

> They used to have all those O'Reilly and EFF stickers, and working at a help desk at the time, it felt very authentic. Then everything got super nice in season 2 -- leather couches, people were dressing nicely, etc. It kind of lost its charm.

That sounds like a pretty realistic allegory of the last two decades in Free Software (or software in general, or the web...)

Re: Fire declared in OVH SBG2 datacentre building

#299
post #264

Most modern data centres wouldn't have had this issue, at least in in Australia they use Argonite suppression systems, these work by using a gas that is a mixture of argon and nitrogen that suppresses fire by depleting oxygen in the the data hall.

I'm seeing quite a lot of repeated sentiment throughout the comments that Halon is illegal and is no longer used. Is the situation "Halon is legal in Australia" or "Halon isn't actually illegal per se if you don't use a lot of it"?

Halon and Argonite are unrelated.

Re: Fire declared in OVH SBG2 datacentre building

#300

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…

I am literally in SBG2 so that has been fun.

Turns out, our disaster recovery plan is pretty good.

Datacenter burned down and I still was up 4 hours later in another data center with zero data loss. Good times.

Post reply on HN