111–120 of 613 posts
Video here. This will be Disaster Recovery 101 material. https://mobile.twitter.com/abonin_DNA/status/136953802824345...
Ouch, that's not pretty... but it seems that the fire was constrained to 1 or 2 sectors (inside the building) - per their updates Not sure how good were the fire suppression systems of the building.
I can't see anything about a fire suppression system mentioned? Doesn't OVH have one, except for colocation datacenters? A fire detection system using eg. lasers and Inergen(or Argonite) for putting the fire out is commonly used in datacenters. The gas fills the room and reduces the amount of oxygen in the room so most fires are put out within a minute. The cool thing is that the gas is designed to be used in rooms w…
Are fires common in data centres? Specialized fire suppression tech seems to indicate that they are.
I can't see anything about a fire suppression system mentioned? Doesn't OVH have one, except for colocation datacenters? A fire detection system using eg. lasers and Inergen(or Argonite) for putting the fire out is commonly used in datacenters. The gas fills the room and reduces the amount of oxygen in the room so most fires are put out within a minute. The cool thing is that the gas is designed to be used in rooms w…
Are fires common in data centres? Specialized fire suppression tech seems to indicate that they are.
Earlier quoted context omitted.
Got burned once (no pun intended), learned my lesson. Hot spare on a different continent with replicated data along with a third box just for backups. The backup box gets offsite backups held in a safe with another redundant copy in another site in another safe. Restores are tested quarterly. Keep backups of backups. Once bitten, twice shy.
> Hot spare on a different continent Just be cautious about data locality laws (not likely to affect you as joe average, more for businesses)
Earlier quoted context omitted.
Just guessing, but maybe a fire suppression system going off could wipe out all the machines? The couple datacenters I've been inside were small, old and used halon gas which wasn't supposed to destroy the machines. No idea how it works in big places these days.
A few years back there was an incident in Sweden where noise coming from the gas based fire suppression system going off destroyed hard drives [1]. 1. https://www.theregister.com/2018/04/26/decibels_destroy_disk...
Earlier quoted context omitted.
One of my backup servers used to be in the same datacenter as the primary server. I only recently moved it to a different host. It's still in the same city, though, so I'm considering other options. I'm not a big fan of just-make-a-tarball-of-everything-and-upload-it-to-the-cloud backup methodology, I prefer something a bit more incremental. But with Backblaze B2 being so cheap, I might as well just upload tarballs t…
I’ve taken to uploading via rsync or similar entire copies - as tarballs use the whole bandwidth each time but rsync on files brings only the changes.
Unfortunately A lot of people are going to find out the hard way today why AWS/GCP/Big Expensive Cloud is so expensive (Hint: they have redundancy and failover procedures which drive up costs). Keep in mind I’m talking not of “downtime” but of actual data loss which might affect business continuity. This is really tragic. I’m hoping they have some kind of multi regional backup/replication and not just multi zones (al…
I use a single server at OVH, and I'm not in the affected DC, but if this DID happen to me I could get back up and running fairly quickly. All our data is mirrored on S3 and off site backups are made frequently enough it wouldn't be an issue.
Plus, you still need to plan for a scenario like this even with AWS or any other cloud provider. It is less likely to happen with those, given the redundancy, but there is still a chance you lose it all without a backup plan.