Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fire declared in OVH SBG2 datacentre building

travaux.ovh.net

Post reply on HN
Print view View original

51–60 of 613 posts

Re: Fire declared in OVH SBG2 datacentre building

#51

Post by akamaka »

worldofmatthew wrote: post #11

https://twitter.com/olesovhcom/status/1369504527544705025 "Update 5:20pm. Everybody is safe. Fire destroyed SBG2. A part of SBG1 is destroyed. Firefighters are protecting SBG3. no impact SBG4."

For some context:

“SBG1, the first Strasbourg data center, consisting of twelve containers, came online in 2012. The 12 containers had a capacity of 12,000 servers.

SBG2 is a non-container data center in 2016 using its “Tower” design with a capacity of 30,000 servers.

SBG3 tower was built in 2017 with a capacity of 30,000 servers.

SBG4 was built in 2013 as several containers to augment capacity, but was decommissioned in 2018 and moved to SB3”

https://baxtel.com/data-center/ovh-strasbourg-campus

Re: Fire declared in OVH SBG2 datacentre building

#52

Post by mhh__ »

meepmorp wrote: post #23
lovedswain wrote: post #15

He posted an update, seems SBG2 is totally destroyed. Ouch https://twitter.com/Onepamopa/status/1369484420982407173

> DID ANYTHING SURVIVE? ANYTHING AT ALL?????? OUR DATA IS IN SBG2. WHAT DO WE DO NOW ?!?! Double check those backups, folks.

One of the things I'm still extremely grateful for is that I learnt the basics of computer science from an ex-oracle guy turned secondary school teacher, who wasn't the best programmer let's say but who absolutely drilled into us (I was probably the only one listening but still) the importance of code quality, backups, information security etc.

Nothing fancy, but it's the kind of bread and butter intuition you need to avoid walking straight off a cliff.

He also let me sit at the back writing a compiler instead of learning VB.Net, top dude

Re: Fire declared in OVH SBG2 datacentre building

#53

Post by mhh__ »

monkeybutton wrote: post #37
meepmorp wrote: post #23

Earlier quoted context omitted.

> DID ANYTHING SURVIVE? ANYTHING AT ALL?????? OUR DATA IS IN SBG2. WHAT DO WE DO NOW ?!?! Double check those backups, folks.

Trust but verify. As a developer it doesn't matter what sysadmins or anyone else says about backups of your data; if you haven't run your DR plan and verified the results, it doesn't exist.

> Trust but verify.

If it's business critical should you even trust at all?

Re: Fire declared in OVH SBG2 datacentre building

#54

Post by AdamJacobMuller »

worldofmatthew wrote: post #11

https://twitter.com/olesovhcom/status/1369504527544705025 "Update 5:20pm. Everybody is safe. Fire destroyed SBG2. A part of SBG1 is destroyed. Firefighters are protecting SBG3. no impact SBG4."

> Fire destroyed SBG2

This is crazy.

SBG2 was HUGE and if this isn't a translation error on the part of Octave (which I could understand given the stress and ESL) I have a hard time fathoming what kind of fire could destroy a whole facility with nearly 1000 racks of equipment spread out across separated halls.

I'm really hoping "destroyed" here means "we lost all power and network core and there's smoke/fire/physical damage to SOME of that"

I can't even fathom a worst-case scenario of a transformer explosion (which does occur and I've seen the aftermath of) having this big of an impact. Datacenters are built to contain and mitigate these kinds of issues. Fire breaks, dry-pipe sprinkler systems and fire-extinguishing gas systems are all designed to prevent a fire from becoming large-scale.

Really glad nobody was hurt. OVH is gonna have a bad time cleaning all this up.

Re: Fire declared in OVH SBG2 datacentre building

#55

Post by mhh__ »

lgleason wrote: post #7

https://www.youtube.com/watch?v=1EBfxjSFAxQ

I'm not sure if it's because my tolerance of Graham Linehan has snapped or not, but I barely laugh at the IT Crowd any more. As with other GL shows I find it's just mostly held together but the cast's delivery and such

The laugh track and the writing is honestly dated even by the standards of Dads Army.

Re: Fire declared in OVH SBG2 datacentre building

#56

Post by dylan604 »

sverhagen wrote: post #41
fy20 wrote: post #39

Earlier quoted context omitted.

I have three servers (1 OVH - different location, 2 DO). The only thing I backup is the DB, which is synced daily to S3. There's a rule to automatically delete files after 30 days to handle GDPR and stop the bucket and costs spiralling out of control. Everything is managed with Ansible and Terraform (on DO side), so I could probably get everything back up and running in less than an hour if needed.

> probably That makes it sound like you didn't try/practice. I imagine that in a real-life scenario things will be a little more painful than in one's imagination.

Exactly. Having a plan is only part of it. Good disaster plans do dry runs a couple of times a year (when time changes is always convenient reminder). If you rehearse the recovery when you're not panicked, you have a better chance of not skipping a step when the timing is much more crucial. Also, some sort of guide with steps given procedurally is a great idea.

Re: Fire declared in OVH SBG2 datacentre building

#57

Post by dylan604 »

bombcar wrote: post #20
kijin wrote: post #14

Earlier quoted context omitted.

One of my backup servers used to be in the same datacenter as the primary server. I only recently moved it to a different host. It's still in the same city, though, so I'm considering other options. I'm not a big fan of just-make-a-tarball-of-everything-and-upload-it-to-the-cloud backup methodology, I prefer something a bit more incremental. But with Backblaze B2 being so cheap, I might as well just upload tarballs t…

I’ve taken to uploading via rsync or similar entire copies - as tarballs use the whole bandwidth each time but rsync on files brings only the changes.

Can't you add only changes to a tar?

Re: Fire declared in OVH SBG2 datacentre building

#59

Post by dfsegoat »

geocrasher wrote: post #10

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…

We test rolling over the entire stack to another AWS DR region (just one we dont normally use) from S3 backups, etc. We do this annually and try to introduce some variations to the scenarios. It takes us about 18 hours realistically.

Documentation / SOPs that have been tested thoroughly by various team members are really important. It helps work out any kinks in interpretation, syntax errors etc.

It does feel a little ridiculous at the time for all the effort involved, but incidents like this show why it's so important.

Re: Fire declared in OVH SBG2 datacentre building

#60

Post by qbasic_forever »

AdamJacobMuller wrote: post #54
worldofmatthew wrote: post #11

https://twitter.com/olesovhcom/status/1369504527544705025 "Update 5:20pm. Everybody is safe. Fire destroyed SBG2. A part of SBG1 is destroyed. Firefighters are protecting SBG3. no impact SBG4."

> Fire destroyed SBG2 This is crazy. SBG2 was HUGE and if this isn't a translation error on the part of Octave (which I could understand given the stress and ESL) I have a hard time fathoming what kind of fire could destroy a whole facility with nearly 1000 racks of equipment spread out across separated halls. I'm really hoping "destroyed" here means "we lost all power and network core and there's smoke/fire/physical…

Just guessing, but maybe a fire suppression system going off could wipe out all the machines?

The couple datacenters I've been inside were small, old and used halon gas which wasn't supposed to destroy the machines. No idea how it works in big places these days.

Post reply on HN