https://twitter.com/olesovhcom/status/1369504527544705025 "Update 5:20pm. Everybody is safe. Fire destroyed SBG2. A part of SBG1 is destroyed. Firefighters are protecting SBG3. no impact SBG4."
[deleted]
Fire declared in OVH SBG2 datacentre building
131–140 of 613 posts
Re: Fire declared in OVH SBG2 datacentre building
#132As an industry we like to think we're transparent, honest, and perhaps even based on merit, but I can't find any of that in this messaging.
"We are currently facing a major incident in our Strasbourg datacentre, with a fire declared in the SBG2 building. Firefighters intervened immediately on the spot but were unable to control the SBG2 fire. As a precautionary measure, the electricity was cut off on the whole site, which impacts all our services at SBG1, SBG2, SBG3 and SBG4. If your production is in Strasbourg, we recommend that you activate your Business Recovery Plan."
Incidents are faced, not caused. It's made clear that they called the fire department as soon as they should have, and they did what they could as well, but in reality, your disaster recovery plan and how well you implemented it is what's really the question now, isn't it?
I think it's far easier to ask whether you've done the right thing, user, than it is to ask why a fire managed to take out an entire facility that was designed to prevent that exact scenario, but only if you're OVH.
Re: Fire declared in OVH SBG2 datacentre building
#133Video here. This will be Disaster Recovery 101 material. https://mobile.twitter.com/abonin_DNA/status/136953802824345...
Or disaster prevention. I'm curious how their fire suppression system failed so spectacularly..
Re: Fire declared in OVH SBG2 datacentre building
#134I can't see anything about a fire suppression system mentioned? Doesn't OVH have one, except for colocation datacenters? A fire detection system using eg. lasers and Inergen(or Argonite) for putting the fire out is commonly used in datacenters. The gas fills the room and reduces the amount of oxygen in the room so most fires are put out within a minute. The cool thing is that the gas is designed to be used in rooms w…
> even destroying harddrives
So loud that it destroys hard drives... That's scary, are people's eardrums much more resistant?
Re: Fire declared in OVH SBG2 datacentre building
#135Photos and video from the scene:
https://www.dna.fr/faits-divers-justice/2021/03/10/strasbour...
Re: Fire declared in OVH SBG2 datacentre building
#136Earlier quoted context omitted.
I encourage you to have a look at the operating income that AWS rakes in. Sure, the amount of expertise, redundancy and breadth of service offerings they provide is worth a markup, but they are also significantly more expensive than they need to be. Thanks to being the leader in an oligopoly, and due to patterns like making network egress unjustifiably expensive to keep you (/your data) from leaving.
I think the question here, then is of subjective value. AWS may charge more for egress, but that’s not high enough for it to be a concern for most clients. A bigger, independent concern is probably that there should be sufficient redundancy, backups and such that allows for business continuity. (Note again that I’m not saying that all companies make full use of these features, but those that care for such things do.…
I've never heard of any data centre burning down (and I work in this industry), so never hearing of an AWS DC burning down isn't really saying anything about AWS.
Re: Fire declared in OVH SBG2 datacentre building
#137This is a literal nightmare for me. I can remember several San Diego fires that threatened the original JohnCompanies datacenter[1] circa mid 2000s and thinking about all of the assets and invested time and care that went into every rack in the facility. Very interested to read the post-mortem here ... even more interested in any actionable takeaways from what is a very rare event ... [1] Castle Access, as it was kno…
[deleted]
Re: Fire declared in OVH SBG2 datacentre building
#138Earlier quoted context omitted.
I encourage you to have a look at the operating income that AWS rakes in. Sure, the amount of expertise, redundancy and breadth of service offerings they provide is worth a markup, but they are also significantly more expensive than they need to be. Thanks to being the leader in an oligopoly, and due to patterns like making network egress unjustifiably expensive to keep you (/your data) from leaving.
I think the question here, then is of subjective value. AWS may charge more for egress, but that’s not high enough for it to be a concern for most clients. A bigger, independent concern is probably that there should be sufficient redundancy, backups and such that allows for business continuity. (Note again that I’m not saying that all companies make full use of these features, but those that care for such things do.…
An availability zone (AZ) in AWS eu-west-2 was flooded by a fire protection system going off within the last year. It absolutely did affect workloads in that AZ. That shouldn't have had a large impact on their customers since AWS promote and make as trivial as is viable multi-AZ architectures.
Put another way: one is guided towards making operational good choices rather than being left to discover them yourself. This is a value proposition of public clouds since it commoditises that specialist knowledge.
Re: Fire declared in OVH SBG2 datacentre building
#139The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…
My recovery plan: tarball & upload to Object Store. I'm going to check out exactly how much replication the OVH object store offers, and see about adding a second geographic location, and maybe even a second provider, tomorrow. (My servers aren't in SBG either - phew!)
If your primary data is on OVH, I'd look at using another company's object store if feasible (S3, B2, etc). If possible, on another payment method. (If you want to be really paranoid, something issued under another legal entity.)
There's a whole class of (mostly non-technical) risks that you solve for when you do this.
If anything happens with your payment method (fails and you don't notice in time; all accounts frozen for investigation), OVH account (hacked, suspended), OVH itself (sudden bankruptcy?), etc, then at least you have _one_ other copy. It's not stuff that's likely to happen, but the cost of planning for it at least as far as "haven't completely lost all my data even if it's going to be a pain to restore" here is relatively minimal.
Re: Fire declared in OVH SBG2 datacentre building
#140Earlier quoted context omitted.
Duplicity is your best bet for incremental backups using B2. I use this for my personal server and it works brilliantly.
I'm using rclone, it works very well for the purpose too.
Ditto. Moved to rclone after having a bunch of random small issues with Duplicity that on their own weren't major but made me lose faith in something that's going to be largely operating unsupervised except for a monthly check-in.