Fire declared in OVH SBG2 datacentre building • Hacker News Board

Re: Fire declared in OVH SBG2 datacentre building

#71

Post by atkbrah » Wed, Mar 10, 2021, 5:54 AM UTC

qbasic_forever wrote: post #60

AdamJacobMuller wrote: post #54

Earlier quoted context omitted.

> Fire destroyed SBG2 This is crazy. SBG2 was HUGE and if this isn't a translation error on the part of Octave (which I could understand given the stress and ESL) I have a hard time fathoming what kind of fire could destroy a whole facility with nearly 1000 racks of equipment spread out across separated halls. I'm really hoping "destroyed" here means "we lost all power and network core and there's smoke/fire/physical…

Just guessing, but maybe a fire suppression system going off could wipe out all the machines? The couple datacenters I've been inside were small, old and used halon gas which wasn't supposed to destroy the machines. No idea how it works in big places these days.

A few years back there was an incident in Sweden where noise coming from the gas based fire suppression system going off destroyed hard drives [1].

1. https://www.theregister.com/2018/04/26/decibels_destroy_disk...

Re: Fire declared in OVH SBG2 datacentre building

#72

Post by walrus01 » Wed, Mar 10, 2021, 5:56 AM UTC

Reminder to not only have backups, but also have some periodic OFFLINE backups.

If your primary is set up with credentials to automatically transfer a copy to the backup destination over the network, what happens if your primary gets pwned and the access is used to encrypt or delete the backup?

Secondly, test doing restores of your backups, and have methods/procedures in place for exactly what a restore looks like.

Re: Fire declared in OVH SBG2 datacentre building

#73

Post by rsync » Wed, Mar 10, 2021, 6:02 AM UTC

This is a literal nightmare for me.

I can remember several San Diego fires that threatened the original JohnCompanies datacenter[1] circa mid 2000s and thinking about all of the assets and invested time and care that went into every rack in the facility.

Very interested to read the post-mortem here ... even more interested in any actionable takeaways from what is a very rare event ...

[1] Castle Access, as it was known, at the intersection of Aero Dr. and the 15 ... was later bought by Redit, then Kio ...

Re: Fire declared in OVH SBG2 datacentre building

#74

Post by axaxs » Wed, Mar 10, 2021, 6:02 AM UTC

AmericanChopper wrote: post #29

tothrowaway wrote: post #24

Earlier quoted context omitted.

I'm at OVH as well (in the BHS datacenter, fortunately). I run my entire production system on one beefy machine. The apps and database are replicated to a backup machine hosted with Hetzner (in their Germany datacenter). I also run a tiny VM at OVH which proxies all traffic to Hetzner. I use a failover IP to point at the big rig at OVH. If the main machine fails, I move the failover IP to the VM, which sends all traf…

> I like knowing that I have a complete backup of my entire business within arm's reach. It could also provide a burglar a fantastic opportunity to pivot into career in data breaches.

LUKS is your friend.

Re: Fire declared in OVH SBG2 datacentre building

#75

Post by ogaj » Wed, Mar 10, 2021, 6:04 AM UTC

Wow. Weren't they just angling for an IPO? I wonder how much of this was insured and what the impact is to their overall operations.

Re: Fire declared in OVH SBG2 datacentre building

#76

Post by tyingq » Wed, Mar 10, 2021, 6:10 AM UTC

nnx wrote: post #43

hashhar wrote: post #31

Earlier quoted context omitted.

AWS and GCP are also prone to same kind of data loss if the AZ you are operating in goes down. They don't automatically geo-replicate things. You still need a backup for the torched EC2 instance to be able to relaunch in another AZ/region.

That's true, but it seems whole of SBG region for OVH is within same disaster radius for one fire... with SBG2 destroyed and SBG1 partly damaged. "The whole site has been isolated, which impacts all our services on SBG1, SBG2, SBG3 and SBG4. " Wonder if those SBGx were advertised as being the same as "Availability Zones" - when other cloud providers ensure zones are distanced enough from each other (~1km at least) to…

The buildings are VERY close to one another.

https://cdn.baxtel.com/data-center/ovh-strasbourg-campus/pho...

Re: Fire declared in OVH SBG2 datacentre building

#77

Post by dylan604 » Wed, Mar 10, 2021, 6:11 AM UTC

Shank wrote: post #67

A status update on the OVH tracker for a different datacenter (LIM-1 / Limburg) says "We are going to intervene in the rack to replace a large number of power supply cables that could have an insulation defect." [0][1] The same type of issue is "planned" in BHS [3] and GRA [2]. Eerie timing: do they possibly suspect some bad cables? [0]: http://travaux.ovh.net/?do=details&id=49016 [1]: http://travaux.ovh.net/?do=deta…

>Eerie timing: do they possibly suspect some bad cables?

Why not? Cables with ratings lower than the load they are carrying is a prime cause for electrical fires. If the load is too high for long enough, the shielding melts away, and if it is close enough for other material to catch fire then that's the ball game. It's a common cause for home electrical fires. Some lamp with poor wiring catches the drapes on fire, etc. Wouldn't think a data center would have flammable curtains though.

Re: Fire declared in OVH SBG2 datacentre building

#78

Post by muglug » Wed, Mar 10, 2021, 6:11 AM UTC

mhh__ wrote: post #55

lgleason wrote: post #7

https://www.youtube.com/watch?v=1EBfxjSFAxQ

I'm not sure if it's because my tolerance of Graham Linehan has snapped or not, but I barely laugh at the IT Crowd any more. As with other GL shows I find it's just mostly held together but the cast's delivery and such The laugh track and the writing is honestly dated even by the standards of Dads Army.

The IT Crowd's comedy became dated incredibly quickly, just like Father Ted's.

Comedies that came later ditched the laugh track. They had to work harder to get viewers at home to laugh, but ultimately a bunch of them (starting with The UK Office) hold up much better as a result.

Re: Fire declared in OVH SBG2 datacentre building

#79

Post by jfrunyon » Wed, Mar 10, 2021, 6:11 AM UTC

geocrasher wrote: post #10

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…

Servers are at a mix of "cloud" providers, and on-site. Most data (including system configs!) is backed up on-site nightly, and to B2 nightly with historical copies - and critical data is also live-replicated to our international branches. (Some "meh" data is backed up only to B2, like our phone logs; we can get most of the info from our carrier anyway).

Our goal and the reason we have a lot of stuff backed up on-prem is to have our most time-critical operations back up within a couple of hours - unless the building is destroyed, in which case that's a moot point and we'll take what we can get.

A dev wiped our almost-monolithic sales/manufacturing/billing/etc MySQL database a month or two ago. (I have been repeatedly overruled on the topic of taking access to prod away from devs) We were down for around an hour. Most of that time was spent pulling gigs of data out of the binlog without also wiping it all again. Because our nightly backups had failed a couple weeks prior - after our most recent monthly "glance at it".

Re: Fire declared in OVH SBG2 datacentre building

#80

Post by Thorrez » Wed, Mar 10, 2021, 6:12 AM UTC

AmericanChopper wrote: post #29

tothrowaway wrote: post #24

Earlier quoted context omitted.

I'm at OVH as well (in the BHS datacenter, fortunately). I run my entire production system on one beefy machine. The apps and database are replicated to a backup machine hosted with Hetzner (in their Germany datacenter). I also run a tiny VM at OVH which proxies all traffic to Hetzner. I use a failover IP to point at the big rig at OVH. If the main machine fails, I move the failover IP to the VM, which sends all traf…

> I like knowing that I have a complete backup of my entire business within arm's reach. It could also provide a burglar a fantastic opportunity to pivot into career in data breaches.

But it does protect somewhat against ransomware on the servers.