Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fire declared in OVH SBG2 datacentre building

travaux.ovh.net

Post reply on HN
Print view View original

41–50 of 613 posts

Re: Fire declared in OVH SBG2 datacentre building

#41

Post by sverhagen »

fy20 wrote: post #39
geocrasher wrote: post #10

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…

I have three servers (1 OVH - different location, 2 DO). The only thing I backup is the DB, which is synced daily to S3. There's a rule to automatically delete files after 30 days to handle GDPR and stop the bucket and costs spiralling out of control. Everything is managed with Ansible and Terraform (on DO side), so I could probably get everything back up and running in less than an hour if needed.

> probably

That makes it sound like you didn't try/practice. I imagine that in a real-life scenario things will be a little more painful than in one's imagination.

Re: Fire declared in OVH SBG2 datacentre building

#42

Post by tialaramex »

geocrasher wrote: post #10

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…

If you are a corporate entity of some kind, the final layer of your plan should always be "Go bankrupt". You can't successfully recover from every possible disaster and you shouldn't try to. In the event of a sufficiently unlikely event, your business fails and every penny spent attempting the impossible will be wasted, move on and let professional administrators salvage what they can for your creditors.

Lots of people plan for specific elements they can imagine and forget other equally or even more important things they are going to need in a disaster. Check out how many organisations that doubtless have 24/7 IT support in case a web server goes down somehow had no plan for what happens if it's unsafe for their 500 call centre employees to sit in tiny cubicles answering phones all day even though pandemic respiratory viruses are so famously likely that Gates listed them consistently as the #1 threat.

Re: Fire declared in OVH SBG2 datacentre building

#43

Post by nnx »

hashhar wrote: post #31
pm90 wrote: post #17

Unfortunately A lot of people are going to find out the hard way today why AWS/GCP/Big Expensive Cloud is so expensive (Hint: they have redundancy and failover procedures which drive up costs). Keep in mind I’m talking not of “downtime” but of actual data loss which might affect business continuity. This is really tragic. I’m hoping they have some kind of multi regional backup/replication and not just multi zones (al…

AWS and GCP are also prone to same kind of data loss if the AZ you are operating in goes down. They don't automatically geo-replicate things. You still need a backup for the torched EC2 instance to be able to relaunch in another AZ/region.

That's true, but it seems whole of SBG region for OVH is within same disaster radius for one fire... with SBG2 destroyed and SBG1 partly damaged.

"The whole site has been isolated, which impacts all our services on SBG1, SBG2, SBG3 and SBG4. "

Wonder if those SBGx were advertised as being the same as "Availability Zones" - when other cloud providers ensure zones are distanced enough from each other (~1km at least) to likely survive events such as fire.

Re: Fire declared in OVH SBG2 datacentre building

#44

Post by gameshot911 »

sparrish wrote: post #38
geocrasher wrote: post #10

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…

Got burned once (no pun intended), learned my lesson. Hot spare on a different continent with replicated data along with a third box just for backups. The backup box gets offsite backups held in a safe with another redundant copy in another site in another safe. Restores are tested quarterly. Keep backups of backups. Once bitten, twice shy.

Have you been bitten, personally? If so, story time?

Re: Fire declared in OVH SBG2 datacentre building

#45

Post by [deleted] »

leajkinUnk wrote: post #36
pm90 wrote: post #17

Unfortunately A lot of people are going to find out the hard way today why AWS/GCP/Big Expensive Cloud is so expensive (Hint: they have redundancy and failover procedures which drive up costs). Keep in mind I’m talking not of “downtime” but of actual data loss which might affect business continuity. This is really tragic. I’m hoping they have some kind of multi regional backup/replication and not just multi zones (al…

“Big cloud” has had fires take out clusters, and somehow they manage to keep it out of the news. In spite of the redundancy and failover procedures, keeping your data centers running when one of the clusters was recently *on fire* is something that is often only possible due to heroic efforts. When I say “heroic efforts”, that’s in contrast to “ordinary error recovery and failover”, which is the way you’d want to han…

[deleted]

Re: Fire declared in OVH SBG2 datacentre building

#46

Post by hashhar »

nnx wrote: post #43
hashhar wrote: post #31

Earlier quoted context omitted.

AWS and GCP are also prone to same kind of data loss if the AZ you are operating in goes down. They don't automatically geo-replicate things. You still need a backup for the torched EC2 instance to be able to relaunch in another AZ/region.

That's true, but it seems whole of SBG region for OVH is within same disaster radius for one fire... with SBG2 destroyed and SBG1 partly damaged. "The whole site has been isolated, which impacts all our services on SBG1, SBG2, SBG3 and SBG4. " Wonder if those SBGx were advertised as being the same as "Availability Zones" - when other cloud providers ensure zones are distanced enough from each other (~1km at least) to…

Thats a fair point. If OVH does market them as AZs then it's disingenuous and liable to suits IMO.

Re: Fire declared in OVH SBG2 datacentre building

#47

Post by sebmellen »

gingerlime wrote: post #32
sebmellen wrote: post #18

Earlier quoted context omitted.

Duplicity is your best bet for incremental backups using B2. I use this for my personal server and it works brilliantly.

I thought so too for a long while. Until I was trying to restore something (just to test things), and wasn’t able to... it might have been specific to our GPG or an older version or something... but I decided to switch to restic and am much happier now. Restic has a single binary that takes care of everything. It feels more modern and seems to work really well. Never had any issue restoring from it. Just one data poi…

Interesting, I will check Restic out, I’ve heard other good things about it. Duplicity is a bit of a pain to set up and Restic’s single binary model is more straightforward (Go is a miracle). Thanks for the recommendation!

GPG is a bit quirky but I do regularly check my backups and restores (if once every few months counts as regular).

Re: Fire declared in OVH SBG2 datacentre building

#48

Post by notaharvardmba »

tothrowaway wrote: post #24
geocrasher wrote: post #10

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…

I'm at OVH as well (in the BHS datacenter, fortunately). I run my entire production system on one beefy machine. The apps and database are replicated to a backup machine hosted with Hetzner (in their Germany datacenter). I also run a tiny VM at OVH which proxies all traffic to Hetzner. I use a failover IP to point at the big rig at OVH. If the main machine fails, I move the failover IP to the VM, which sends all traf…

Bacula has some really cool features for cloud backups.

https://bacula.org

Re: Fire declared in OVH SBG2 datacentre building

#50

Post by the_duke »

pm90 wrote: post #17

Unfortunately A lot of people are going to find out the hard way today why AWS/GCP/Big Expensive Cloud is so expensive (Hint: they have redundancy and failover procedures which drive up costs). Keep in mind I’m talking not of “downtime” but of actual data loss which might affect business continuity. This is really tragic. I’m hoping they have some kind of multi regional backup/replication and not just multi zones (al…

I encourage you to have a look at the operating income that AWS rakes in.

Sure, the amount of expertise, redundancy and breadth of service offerings they provide is worth a markup, but they are also significantly more expensive than they need to be.

Thanks to being the leader in an oligopoly, and due to patterns like making network egress unjustifiably expensive to keep you (/your data) from leaving.

Post reply on HN