Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fire declared in OVH SBG2 datacentre building

travaux.ovh.net

Post reply on HN
Print view View original

31–40 of 613 posts

Re: Fire declared in OVH SBG2 datacentre building

#31

Post by hashhar »

pm90 wrote: post #17

Unfortunately A lot of people are going to find out the hard way today why AWS/GCP/Big Expensive Cloud is so expensive (Hint: they have redundancy and failover procedures which drive up costs). Keep in mind I’m talking not of “downtime” but of actual data loss which might affect business continuity. This is really tragic. I’m hoping they have some kind of multi regional backup/replication and not just multi zones (al…

AWS and GCP are also prone to same kind of data loss if the AZ you are operating in goes down.

They don't automatically geo-replicate things. You still need a backup for the torched EC2 instance to be able to relaunch in another AZ/region.

Re: Fire declared in OVH SBG2 datacentre building

#32

Post by gingerlime »

sebmellen wrote: post #18
kijin wrote: post #14

Earlier quoted context omitted.

One of my backup servers used to be in the same datacenter as the primary server. I only recently moved it to a different host. It's still in the same city, though, so I'm considering other options. I'm not a big fan of just-make-a-tarball-of-everything-and-upload-it-to-the-cloud backup methodology, I prefer something a bit more incremental. But with Backblaze B2 being so cheap, I might as well just upload tarballs t…

Duplicity is your best bet for incremental backups using B2. I use this for my personal server and it works brilliantly.

I thought so too for a long while. Until I was trying to restore something (just to test things), and wasn’t able to... it might have been specific to our GPG or an older version or something... but I decided to switch to restic and am much happier now.

Restic has a single binary that takes care of everything. It feels more modern and seems to work really well. Never had any issue restoring from it.

Just one data point. Stick to whatever works for you. But important to test not only your backups, but also restores!

Re: Fire declared in OVH SBG2 datacentre building

#33

Post by baggy_trough »

AmericanChopper wrote: post #29
tothrowaway wrote: post #24

Earlier quoted context omitted.

I'm at OVH as well (in the BHS datacenter, fortunately). I run my entire production system on one beefy machine. The apps and database are replicated to a backup machine hosted with Hetzner (in their Germany datacenter). I also run a tiny VM at OVH which proxies all traffic to Hetzner. I use a failover IP to point at the big rig at OVH. If the main machine fails, I move the failover IP to the VM, which sends all traf…

> I like knowing that I have a complete backup of my entire business within arm's reach. It could also provide a burglar a fantastic opportunity to pivot into career in data breaches.

It all depends on your paranoia level of data hacking burglars vs. vaporized data centers.

Re: Fire declared in OVH SBG2 datacentre building

#34

Post by geocrasher »

sebmellen wrote: post #18
kijin wrote: post #14

Earlier quoted context omitted.

One of my backup servers used to be in the same datacenter as the primary server. I only recently moved it to a different host. It's still in the same city, though, so I'm considering other options. I'm not a big fan of just-make-a-tarball-of-everything-and-upload-it-to-the-cloud backup methodology, I prefer something a bit more incremental. But with Backblaze B2 being so cheap, I might as well just upload tarballs t…

Duplicity is your best bet for incremental backups using B2. I use this for my personal server and it works brilliantly.

I'm using rclone, it works very well for the purpose too.

Re: Fire declared in OVH SBG2 datacentre building

#36

Post by leajkinUnk »

pm90 wrote: post #17

Unfortunately A lot of people are going to find out the hard way today why AWS/GCP/Big Expensive Cloud is so expensive (Hint: they have redundancy and failover procedures which drive up costs). Keep in mind I’m talking not of “downtime” but of actual data loss which might affect business continuity. This is really tragic. I’m hoping they have some kind of multi regional backup/replication and not just multi zones (al…

“Big cloud” has had fires take out clusters, and somehow they manage to keep it out of the news. In spite of the redundancy and failover procedures, keeping your data centers running when one of the clusters was recently *on fire* is something that is often only possible due to heroic efforts.

When I say “heroic efforts”, that’s in contrast to “ordinary error recovery and failover”, which is the way you’d want to handle a DC fire, because DC fires happen often enough.

The thing is, while these big companies have a much larger base of expertise to draw on and simply more staff time to throw at problems, there are factors which incentivize these employees to *increase risk* rather than reduce it.

These big companies put pressure on all their engineers to figure out ways to drive down costs. So, while a big cloud provider won’t make a rookie mistake—they won’t forget to run disaster recovery drills, they won’t forget to make backups and run test restores—they *will* do a bunch of calculations to figure out how close to disaster they can run in order to save money. The real disaster will then reveal some false, hidden assumption in their error recovery models.

Or in other words, the big companies solve all the easy problems and then create new, hard problems.

Re: Fire declared in OVH SBG2 datacentre building

#37

Post by monkeybutton »

meepmorp wrote: post #23
lovedswain wrote: post #15

He posted an update, seems SBG2 is totally destroyed. Ouch https://twitter.com/Onepamopa/status/1369484420982407173

> DID ANYTHING SURVIVE? ANYTHING AT ALL?????? OUR DATA IS IN SBG2. WHAT DO WE DO NOW ?!?! Double check those backups, folks.

Trust but verify. As a developer it doesn't matter what sysadmins or anyone else says about backups of your data; if you haven't run your DR plan and verified the results, it doesn't exist.

Re: Fire declared in OVH SBG2 datacentre building

#38

Post by sparrish »

geocrasher wrote: post #10

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…

Got burned once (no pun intended), learned my lesson.

Hot spare on a different continent with replicated data along with a third box just for backups. The backup box gets offsite backups held in a safe with another redundant copy in another site in another safe.

Restores are tested quarterly.

Keep backups of backups. Once bitten, twice shy.

Re: Fire declared in OVH SBG2 datacentre building

#39

Post by fy20 »

geocrasher wrote: post #10

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…

I have three servers (1 OVH - different location, 2 DO). The only thing I backup is the DB, which is synced daily to S3. There's a rule to automatically delete files after 30 days to handle GDPR and stop the bucket and costs spiralling out of control.

Everything is managed with Ansible and Terraform (on DO side), so I could probably get everything back up and running in less than an hour if needed.

Re: Fire declared in OVH SBG2 datacentre building

#40

Post by tgsovlerkhgsel »

AmericanChopper wrote: post #29
tothrowaway wrote: post #24

Earlier quoted context omitted.

I'm at OVH as well (in the BHS datacenter, fortunately). I run my entire production system on one beefy machine. The apps and database are replicated to a backup machine hosted with Hetzner (in their Germany datacenter). I also run a tiny VM at OVH which proxies all traffic to Hetzner. I use a failover IP to point at the big rig at OVH. If the main machine fails, I move the failover IP to the VM, which sends all traf…

> I like knowing that I have a complete backup of my entire business within arm's reach. It could also provide a burglar a fantastic opportunity to pivot into career in data breaches.

This problem is usually solved through encryption.
Post reply on HN