Live data from Hacker News

Fire declared in OVH SBG2 datacentre building

travaux.ovh.net

231–240 of 613 posts

Re: Fire declared in OVH SBG2 datacentre building

#231
post #200

Earlier quoted context omitted.

I'm at OVH as well (in the BHS datacenter, fortunately). I run my entire production system on one beefy machine. The apps and database are replicated to a backup machine hosted with Hetzner (in their Germany datacenter). I also run a tiny VM at OVH which proxies all traffic to Hetzner. I use a failover IP to point at the big rig at OVH. If the main machine fails, I move the failover IP to the VM, which sends all traf…

Are you truly paranoid? If my money and/or job depended on having something running without (or with minimal) disruption I would be as paranoid as you, too. BTW - Some people call this business recovery plan, not plain paranoia ;-)

It's not paranoia if you're right. All of the risks GP is protecting against are things that happen to someone every day, and they should be seen like wearing the seat belt in a car.

Re: Fire declared in OVH SBG2 datacentre building

#232

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…

Pictures from the fire: https://www.dna.fr/amp/faits-divers-justice/2021/03/10/stras...

And here's some more from firefighters, while it was burning:

https://twitter.com/xgarreau/status/1369559995491172354

Looks glowing red to me.

Re: Fire declared in OVH SBG2 datacentre building

#233
post #53

Earlier quoted context omitted.

Trust but verify. As a developer it doesn't matter what sysadmins or anyone else says about backups of your data; if you haven't run your DR plan and verified the results, it doesn't exist.

> Trust but verify. If it's business critical should you even trust at all?

yeah backups are a case of "don't trust, verify"

Re: Fire declared in OVH SBG2 datacentre building

#234
Been there, done that.

The 2nd worst thing, IF you happen to catch it soon and control it, is that the temp raise triggers many alerts and automatic controls.

So when it's controlled, it's still a real nightmare. Here, firemen could not even control it...

Re: Fire declared in OVH SBG2 datacentre building

#237

Earlier quoted context omitted.

This problem is usually solved through encryption.

If I were to ask my CISO if I was allowed to bring the production database home, I’m pretty sure his answer wouldn’t be “as long as you encrypt it”.

That's because he doesn't trust you with this data. That has nothing to do with encryption safety.

There is nothing magical about data centers making them safe while your local copy isn't.

Re: Fire declared in OVH SBG2 datacentre building

#238
post #146

According to the official status page, the whole datacenter is still green http://status.ovh.com/vms/index_sbg2.html

I feel like there should be place to report infrastructure suppliers with misleading status pages, some kind of crowdsourced database. Without this information, you only find out that they are misleading when something goes very wrong.

At best you might be missing out on some SLA refunds, but at worst it could be disasterous for a business. I've been on the wrong side of a update-by-hand status system from a hosting provider before and it wasn't fun.

Re: Fire declared in OVH SBG2 datacentre building

#239
post #216

Earlier quoted context omitted.

I think they announced their IP plans yesterday [1], which is probably the worst timing one can have (if there even is a good timing for a datacenter burning down, probably there isn't). If they have a good insurance I'm confident this will have little impact on their operations, I really hope they do. I host a few components on OVH/SoYouStart dedicated servers, luckily not mission critical, but still had rather good…

The publicity damage alone will be on par with (if not bigger) their replacement costs. I wouldn’t be surprised if they had to rebrand. The timing is so bad that it becomes almost suspicious. When is the best time to sabotage your competitor? When they are the most visible.

SBG is a small part of their operations. I doubt it will have a lasting issue on their image

Re: Fire declared in OVH SBG2 datacentre building

#240
post #214

Some pictures of the building with firefighters at work https://www.dna.fr/faits-divers-justice/2021/03/10/strasbour... Edit: Video at https://www.youtube.com/watch?v=a9jL_THG58U Satellite view of the site on Google Maps https://goo.gl/maps/L2T6YNFCtiyDdiNv7

Easy to see now that a lightly constructed five story cube might not be a fully fire proof.

How do you see that?
Post reply on HN