Earlier quoted context omitted.
I'm at OVH as well (in the BHS datacenter, fortunately). I run my entire production system on one beefy machine. The apps and database are replicated to a backup machine hosted with Hetzner (in their Germany datacenter). I also run a tiny VM at OVH which proxies all traffic to Hetzner. I use a failover IP to point at the big rig at OVH. If the main machine fails, I move the failover IP to the VM, which sends all traf…
Are you truly paranoid? If my money and/or job depended on having something running without (or with minimal) disruption I would be as paranoid as you, too. BTW - Some people call this business recovery plan, not plain paranoia ;-)
Fire declared in OVH SBG2 datacentre building
231–240 of 613 posts
Re: Fire declared in OVH SBG2 datacentre building
#232The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…
Pictures from the fire: https://www.dna.fr/amp/faits-divers-justice/2021/03/10/stras...
https://twitter.com/xgarreau/status/1369559995491172354
Looks glowing red to me.
Re: Fire declared in OVH SBG2 datacentre building
#233Earlier quoted context omitted.
Trust but verify. As a developer it doesn't matter what sysadmins or anyone else says about backups of your data; if you haven't run your DR plan and verified the results, it doesn't exist.
> Trust but verify. If it's business critical should you even trust at all?
Re: Fire declared in OVH SBG2 datacentre building
#234The 2nd worst thing, IF you happen to catch it soon and control it, is that the temp raise triggers many alerts and automatic controls.
So when it's controlled, it's still a real nightmare. Here, firemen could not even control it...
Re: Fire declared in OVH SBG2 datacentre building
#235Earlier quoted context omitted.
With water, they said.
Liquid cooling.
Re: Fire declared in OVH SBG2 datacentre building
#236I don’t know what ovh is and going to the site point me to a speed test with no information.
Largest hosting provider in Europe, probably top10 or top5 in the world.
Re: Fire declared in OVH SBG2 datacentre building
#237Earlier quoted context omitted.
This problem is usually solved through encryption.
If I were to ask my CISO if I was allowed to bring the production database home, I’m pretty sure his answer wouldn’t be “as long as you encrypt it”.
There is nothing magical about data centers making them safe while your local copy isn't.
Re: Fire declared in OVH SBG2 datacentre building
#238According to the official status page, the whole datacenter is still green http://status.ovh.com/vms/index_sbg2.html
At best you might be missing out on some SLA refunds, but at worst it could be disasterous for a business. I've been on the wrong side of a update-by-hand status system from a hosting provider before and it wasn't fun.
Re: Fire declared in OVH SBG2 datacentre building
#239Earlier quoted context omitted.
I think they announced their IP plans yesterday [1], which is probably the worst timing one can have (if there even is a good timing for a datacenter burning down, probably there isn't). If they have a good insurance I'm confident this will have little impact on their operations, I really hope they do. I host a few components on OVH/SoYouStart dedicated servers, luckily not mission critical, but still had rather good…
The publicity damage alone will be on par with (if not bigger) their replacement costs. I wouldn’t be surprised if they had to rebrand. The timing is so bad that it becomes almost suspicious. When is the best time to sabotage your competitor? When they are the most visible.
Re: Fire declared in OVH SBG2 datacentre building
#240Some pictures of the building with firefighters at work https://www.dna.fr/faits-divers-justice/2021/03/10/strasbour... Edit: Video at https://www.youtube.com/watch?v=a9jL_THG58U Satellite view of the site on Google Maps https://goo.gl/maps/L2T6YNFCtiyDdiNv7
Easy to see now that a lightly constructed five story cube might not be a fully fire proof.