Fire declared in OVH SBG2 datacentre building • Hacker News Board

geocrasher wrote: post #10

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…

My disaster recovery plan: we shall rebuild.

Re: Fire declared in OVH SBG2 datacentre building

#14

Post by kijin » Wed, Mar 10, 2021, 4:32 AM UTC

geocrasher wrote: post #10

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…

One of my backup servers used to be in the same datacenter as the primary server. I only recently moved it to a different host. It's still in the same city, though, so I'm considering other options. I'm not a big fan of just-make-a-tarball-of-everything-and-upload-it-to-the-cloud backup methodology, I prefer something a bit more incremental. But with Backblaze B2 being so cheap, I might as well just upload tarballs to B2. As long as I have the data, the servers can be redeployed in a couple of hours at most.

The SBG fire illustrates the importance of geographical redundancy. Just because the datacenters have different numbers at the end doesn't mean that they won't fail at the same time. Apart from a large fire or power outage, there are lots of things that can take out several datacenters in close vicinity at the same time, such as hurricanes and earthquakes.

Re: Fire declared in OVH SBG2 datacentre building

#15

Post by lovedswain » Wed, Mar 10, 2021, 4:33 AM UTC

He posted an update, seems SBG2 is totally destroyed.

Ouch https://twitter.com/Onepamopa/status/1369484420982407173

Re: Fire declared in OVH SBG2 datacentre building

#16

Post by edoceo » Wed, Mar 10, 2021, 4:36 AM UTC

ahmedalsudani wrote: post #13

geocrasher wrote: post #10

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…

My disaster recovery plan: we shall rebuild.

F

Re: Fire declared in OVH SBG2 datacentre building

#17

Post by pm90 » Wed, Mar 10, 2021, 4:37 AM UTC

Unfortunately A lot of people are going to find out the hard way today why AWS/GCP/Big Expensive Cloud is so expensive (Hint: they have redundancy and failover procedures which drive up costs).

Keep in mind I’m talking not of “downtime” but of actual data loss which might affect business continuity.

This is really tragic. I’m hoping they have some kind of multi regional backup/replication and not just multi zones (although from the twitters it appears that only one of the zones was destroyed however the others don’t seem to be operational atm).

Re: Fire declared in OVH SBG2 datacentre building

#18

Post by sebmellen » Wed, Mar 10, 2021, 4:38 AM UTC

kijin wrote: post #14

geocrasher wrote: post #10

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…

One of my backup servers used to be in the same datacenter as the primary server. I only recently moved it to a different host. It's still in the same city, though, so I'm considering other options. I'm not a big fan of just-make-a-tarball-of-everything-and-upload-it-to-the-cloud backup methodology, I prefer something a bit more incremental. But with Backblaze B2 being so cheap, I might as well just upload tarballs t…

Duplicity is your best bet for incremental backups using B2. I use this for my personal server and it works brilliantly.

Re: Fire declared in OVH SBG2 datacentre building

#19

Post by cookiengineer » Wed, Mar 10, 2021, 4:40 AM UTC

bigiain wrote: post #8

There is no cloud, there is just other people’s computers. And they’re on fire.

Ah, that explains why there are so many dark patterns in the cloud! /s

Re: Fire declared in OVH SBG2 datacentre building

#20

Post by bombcar » Wed, Mar 10, 2021, 4:40 AM UTC

kijin wrote: post #14

geocrasher wrote: post #10

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…

One of my backup servers used to be in the same datacenter as the primary server. I only recently moved it to a different host. It's still in the same city, though, so I'm considering other options. I'm not a big fan of just-make-a-tarball-of-everything-and-upload-it-to-the-cloud backup methodology, I prefer something a bit more incremental. But with Backblaze B2 being so cheap, I might as well just upload tarballs t…

I’ve taken to uploading via rsync or similar entire copies - as tarballs use the whole bandwidth each time but rsync on files brings only the changes.