Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fire declared in OVH SBG2 datacentre building

travaux.ovh.net

Post reply on HN
Print view View original

21–30 of 613 posts

Re: Fire declared in OVH SBG2 datacentre building

#21

Post by iamd3vil »

kijin wrote: post #14
geocrasher wrote: post #10

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…

One of my backup servers used to be in the same datacenter as the primary server. I only recently moved it to a different host. It's still in the same city, though, so I'm considering other options. I'm not a big fan of just-make-a-tarball-of-everything-and-upload-it-to-the-cloud backup methodology, I prefer something a bit more incremental. But with Backblaze B2 being so cheap, I might as well just upload tarballs t…

Another option for incremental backups is Restic [0]. It has support to backup to Backblaze B2, Amazon S3 and lots of other places.

[0] https://restic.net/

Re: Fire declared in OVH SBG2 datacentre building

#24

Post by tothrowaway »

geocrasher wrote: post #10

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…

I'm at OVH as well (in the BHS datacenter, fortunately). I run my entire production system on one beefy machine. The apps and database are replicated to a backup machine hosted with Hetzner (in their Germany datacenter). I also run a tiny VM at OVH which proxies all traffic to Hetzner. I use a failover IP to point at the big rig at OVH. If the main machine fails, I move the failover IP to the VM, which sends all traffic to Hetzner.

If OVH is totally down, and the fail over IP doesn't work, I have a fairly low TTL on the DNS.

I backup the database state to S3 every day.

Since I'm truly paranoid, I have an Intel NUC at my house that also replicates the DB. I like knowing that I have a complete backup of my entire business within arm's reach.

Re: Fire declared in OVH SBG2 datacentre building

#25

Post by ethanpil »

geocrasher wrote: post #10

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…

Does anyone have experience with lvarchive or FSArchiver (or similar) to backup images of live systems instead of file based backup solutions?

Re: Fire declared in OVH SBG2 datacentre building

#26

Post by jen20 »

worldofmatthew wrote: post #11

https://twitter.com/olesovhcom/status/1369504527544705025 "Update 5:20pm. Everybody is safe. Fire destroyed SBG2. A part of SBG1 is destroyed. Firefighters are protecting SBG3. no impact SBG4."

An easy mistake (in the quoted tweet) to make given the circumstances, but it probably should be 5:20am rather than pm.

Re: Fire declared in OVH SBG2 datacentre building

#27

Post by jbeales »

geocrasher wrote: post #10

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…

My recovery plan: tarball & upload to Object Store. I'm going to check out exactly how much replication the OVH object store offers, and see about adding a second geographic location, and maybe even a second provider, tomorrow.

(My servers aren't in SBG either - phew!)

Re: Fire declared in OVH SBG2 datacentre building

#28

Post by juangacovas »

geocrasher wrote: post #10

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…

Less than a day for disaster recovery on fresh hardware? Same as my case. As you say, good enough for most purposes, but I'm also looking for improvement. I have offsite realtime replicas for data and mariaDBs, and offsite nightly backups (combo of rsnapshot, lsyncd, mariaDB multi-source replication, and a post-new-install script that setups almost everything in case you have to recover on bare-metal, i.e. no available VM snapshots).

Currently trying to reduce that "less than a day" though. Recently discovered "ReaR" (Relax and Recover) from RedHat and sounds really nice for bare-metal servers. Not everybody runs on virtualized/cloud (being able to recover from VM snapshots is really a plus). Let's share experiencies :)

Re: Fire declared in OVH SBG2 datacentre building

#29

Post by AmericanChopper »

tothrowaway wrote: post #24
geocrasher wrote: post #10

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…

I'm at OVH as well (in the BHS datacenter, fortunately). I run my entire production system on one beefy machine. The apps and database are replicated to a backup machine hosted with Hetzner (in their Germany datacenter). I also run a tiny VM at OVH which proxies all traffic to Hetzner. I use a failover IP to point at the big rig at OVH. If the main machine fails, I move the failover IP to the VM, which sends all traf…

> I like knowing that I have a complete backup of my entire business within arm's reach.

It could also provide a burglar a fantastic opportunity to pivot into career in data breaches.

Post reply on HN