Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fire declared in OVH SBG2 datacentre building

travaux.ovh.net

Post reply on HN
Print view View original

171–180 of 613 posts

Re: Fire declared in OVH SBG2 datacentre building

#171

Post by Scoundreller »

akamaka wrote: post #51
worldofmatthew wrote: post #11

https://twitter.com/olesovhcom/status/1369504527544705025 "Update 5:20pm. Everybody is safe. Fire destroyed SBG2. A part of SBG1 is destroyed. Firefighters are protecting SBG3. no impact SBG4."

For some context: “SBG1, the first Strasbourg data center, consisting of twelve containers, came online in 2012. The 12 containers had a capacity of 12,000 servers. SBG2 is a non-container data center in 2016 using its “Tower” design with a capacity of 30,000 servers. SBG3 tower was built in 2017 with a capacity of 30,000 servers. SBG4 was built in 2013 as several containers to augment capacity, but was decommissione…

and a 2017 (!) article where they were planning to remove 1 and 2 because of power issues!

> OVH to Disassemble Container Data Centers after Epic Outage in Europe

> “This is probably the worst-case scenario that could have happened to us.”

> OVH [...] is planning to shut down and disassemble two of the three data centers on its campus in Strasbourg, France, following a power outage that brought down the entire campus Friday, causing prolonged disruption to customer applications that lasted throughout the day and well into the evening.

https://www.datacenterknowledge.com/uptime/ovh-disassemble-c...

Re: Fire declared in OVH SBG2 datacentre building

#172

Post by torh »

NorwegianDude wrote: post #90

I can't see anything about a fire suppression system mentioned? Doesn't OVH have one, except for colocation datacenters? A fire detection system using eg. lasers and Inergen(or Argonite) for putting the fire out is commonly used in datacenters. The gas fills the room and reduces the amount of oxygen in the room so most fires are put out within a minute. The cool thing is that the gas is designed to be used in rooms w…

A few years ago the company I work for installed suppressors on the Inergen system. It did trigger from time to time, which was tracked to the humidifiers. And yes -- it did destroy harddrives because of the pressure/sound waves before we installed the supressors. Haven't had any incidents after we fixed the humidifiers.

But Inergen (and other gas) is more or less useless if you allow it to escape too quickly. So the cooling system should be a fairly closed circut.

Edit: I'm also a Norwegian dude. :)

Re: Fire declared in OVH SBG2 datacentre building

#173

Post by ikiris »

AdamJacobMuller wrote: post #54
worldofmatthew wrote: post #11

https://twitter.com/olesovhcom/status/1369504527544705025 "Update 5:20pm. Everybody is safe. Fire destroyed SBG2. A part of SBG1 is destroyed. Firefighters are protecting SBG3. no impact SBG4."

> Fire destroyed SBG2 This is crazy. SBG2 was HUGE and if this isn't a translation error on the part of Octave (which I could understand given the stress and ESL) I have a hard time fathoming what kind of fire could destroy a whole facility with nearly 1000 racks of equipment spread out across separated halls. I'm really hoping "destroyed" here means "we lost all power and network core and there's smoke/fire/physical…

This was effectively posted on the outages list as well by someone trustworthy. The pictures also look pretty bad from the outside.

Re: Fire declared in OVH SBG2 datacentre building

#174

Post by Cthulhu_ »

geocrasher wrote: post #10

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…

Personal: I run a webserver for some website (wordpress + xenforo), I've set up a cronjob that creates a backup of /var/www, /etc and a mysql database dump, then uploads it to an S3 bucket (with automatic Glacier archiving after X period set up). It should be fairly straightforward to rent a new server and set things back up. I still dislike having to set up a webserver + php manually though, I don't get why that hasn't been streamlined yet.

My employer has a single rack of servers at HQ. It's positioned at a very specific angle with an AC unit facing it, their exact positions are marked out on the floor in tape. The servers contain VMs that most employees work on, our git repository, issue trackers, and probably customer admin as well. They say they do off-site backups, but honestly, when (not if) that thing goes it'll be a pretty serious impact on the business. They don't like people keeping their code on their take-home laptop either (I can't fathom how my colleagues work and how they can stand working in a large codebase using barebones vim over ssh), but I've employed some professional disobedience there.

Re: Fire declared in OVH SBG2 datacentre building

#175

Post by sofixa »

AdamJacobMuller wrote: post #54
worldofmatthew wrote: post #11

https://twitter.com/olesovhcom/status/1369504527544705025 "Update 5:20pm. Everybody is safe. Fire destroyed SBG2. A part of SBG1 is destroyed. Firefighters are protecting SBG3. no impact SBG4."

> Fire destroyed SBG2 This is crazy. SBG2 was HUGE and if this isn't a translation error on the part of Octave (which I could understand given the stress and ESL) I have a hard time fathoming what kind of fire could destroy a whole facility with nearly 1000 racks of equipment spread out across separated halls. I'm really hoping "destroyed" here means "we lost all power and network core and there's smoke/fire/physical…

OVH design their own datacenters, so it's possible that they missed something or some system or another didn't work as intended, thus the heavy damage.

Re: Fire declared in OVH SBG2 datacentre building

#176

Post by teddyh »

dylan604 wrote: post #57
bombcar wrote: post #20

Earlier quoted context omitted.

I’ve taken to uploading via rsync or similar entire copies - as tarballs use the whole bandwidth each time but rsync on files brings only the changes.

Can't you add only changes to a tar?

Indeed it does; see the --listed-incremental and --incremental options:

https://www.gnu.org/software/tar/manual/tar.html#Incremental...

Re: Fire declared in OVH SBG2 datacentre building

#178

Post by ikiris »

meepmorp wrote: post #23
lovedswain wrote: post #15

He posted an update, seems SBG2 is totally destroyed. Ouch https://twitter.com/Onepamopa/status/1369484420982407173

> DID ANYTHING SURVIVE? ANYTHING AT ALL?????? OUR DATA IS IN SBG2. WHAT DO WE DO NOW ?!?! Double check those backups, folks.

what do we do? Well for starters don't rely on a single geographic location chuckles

Re: Fire declared in OVH SBG2 datacentre building

#180

Post by rgj »

sofixa wrote: post #175
AdamJacobMuller wrote: post #54

Earlier quoted context omitted.

> Fire destroyed SBG2 This is crazy. SBG2 was HUGE and if this isn't a translation error on the part of Octave (which I could understand given the stress and ESL) I have a hard time fathoming what kind of fire could destroy a whole facility with nearly 1000 racks of equipment spread out across separated halls. I'm really hoping "destroyed" here means "we lost all power and network core and there's smoke/fire/physical…

OVH design their own datacenters, so it's possible that they missed something or some system or another didn't work as intended, thus the heavy damage.

They did not have a fire suppression system, only smoke detection. So yeah, they missed something.
Post reply on HN