Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fire declared in OVH SBG2 datacentre building

travaux.ovh.net

Post reply on HN
Print view View original

311–320 of 613 posts

Re: Fire declared in OVH SBG2 datacentre building

#311

Post by dragonwriter »

brmgb wrote: post #262
tialaramex wrote: post #42

Earlier quoted context omitted.

If you are a corporate entity of some kind, the final layer of your plan should always be "Go bankrupt". You can't successfully recover from every possible disaster and you shouldn't try to. In the event of a sufficiently unlikely event, your business fails and every penny spent attempting the impossible will be wasted, move on and let professional administrators salvage what they can for your creditors. Lots of peop…

"Go bankrupt" is not a plan. Becoming insolvent might be the end result of a situation but it's not going to help you deal with it. Let's take an example which might lead to bankruptcy. A typical answer to a major disaster (let's say your main and sole building burning as a typical case) for an SME would be to cease activity, furlough employes and stop or defer every payments you can while you claim insurance and ass…

> "Go bankrupt" is not a plan.

Yes it is. (Though it's better, as GP suggested, as a final layer of a plan and not the only layer.)

> Becoming insolvent might be the end result of a situation but it's not going to help you deal with it.

Insolvency isn't bankruptcy. Becoming insolvent is a consequence, sure. Bankruptcy absolutely does help you deal with that impact, that's rather the point of it.

Re: Fire declared in OVH SBG2 datacentre building

#312

Post by divingdragon »

exikyut wrote: post #281
termau wrote: post #264

Most modern data centres wouldn't have had this issue, at least in in Australia they use Argonite suppression systems, these work by using a gas that is a mixture of argon and nitrogen that suppresses fire by depleting oxygen in the the data hall.

I'm seeing quite a lot of repeated sentiment throughout the comments that Halon is illegal and is no longer used. Is the situation "Halon is legal in Australia" or "Halon isn't actually illegal per se if you don't use a lot of it"?

Halons are being phased out because, being CFCs, they deplete the ozone layer.

Other gases like argon or FM200 are not Halons.

Re: Fire declared in OVH SBG2 datacentre building

#313

Post by Ploskin »

aetherspawn wrote: post #306
bootloop wrote: post #295

Earlier quoted context omitted.

If there is a SLA with consequences associated with it every status page is going to be a lie.

Well, it sucks to catch fire and I care for the employees and the firemen, but if their status page is a lie then I have a whole lot less sympathy for the business. That's shady business and they should feel bad. I can appreciate an honest mistake though, like the status page server cron is hosted in the same cluster that caught fire and hence it burnt down and can't update the page anymore.

Is the status page relevant though? At the very least, OVH immediately made a status announcement on their support page and they've been active on Twitter. I don't see anything shady here. From their support page:

> The whole site has been isolated, which impacts all our services on SBG1, SBG2, SBG3 and SBG4. If your production is in Strasbourg, we recommend to activate your Disaster Recovery Plan

What more could you want?

Re: Fire declared in OVH SBG2 datacentre building

#314

Post by lenartowski »

mwcampbell wrote: post #309

I just recently started moving some services for my business to one of OVH's US-based data centers. Should I take this fire as evidence that OVH is incompetent and get out? I really don't want AWS, or the big three hyperscalers in general, to be the only option.

IMO you should take this fire as evidence, that you need to have (working!) backups wherever you host your data. AWS, GCP Azure are not fire resistant, same as OVH. I don't know if OVH is more or less competent than big three, I choose to trust no one.

Re: Fire declared in OVH SBG2 datacentre building

#315

Post by tweetle_beetle »

vntok wrote: post #274
tweetle_beetle wrote: post #238

Earlier quoted context omitted.

I feel like there should be place to report infrastructure suppliers with misleading status pages, some kind of crowdsourced database. Without this information, you only find out that they are misleading when something goes very wrong. At best you might be missing out on some SLA refunds, but at worst it could be disasterous for a business. I've been on the wrong side of a update-by-hand status system from a hosting…

https://downdetector.fr/

Thanks, can't belive it's taken me 8 years to learn about that.

Re: Fire declared in OVH SBG2 datacentre building

#316

Post by AmericanChopper »

brmgb wrote: post #308
AmericanChopper wrote: post #260

Earlier quoted context omitted.

> There is nothing magical about data centers making them safe while your local copy isn't. Is this a serious comment? My house is not certified as being compliant with any security standards. Here's the list that the 3rd party datacenter we use is certified as complaint with: https://aws.amazon.com/compliance/programs/ The data centers we operate ourselves are audited against several of those standards too. I guess…

Of course it's serious. You are entierly missing the point by quoting the compliance programs followed by AWS whose sole business is being a third party hoster. For most business, what you call sensitive data is customers and orders listing, payment history, inventory if you are dealing in physical goods and HR related files. These are not state secrets. Encryption and a modicum of physical security go a long way. I…

This is quite a significant revision to your previous statement that there’s nothing about a data center that makes it more secure than your house.

This attitude that your data isn’t very important, so it’s fine to not be very concerned about it’s security, while not entirely uncommon, is something most organisations try to avoid when choosing vendors. It’s something consumers are generally unconcerned about, until a breach occurs, and The Intercept write an article about it. At which point I’m sure all the people ITT who are saying it’s fine to take your production database home would be piling on with how stupid the company was for doing ridiculous things like taking a copy of their production database home.

Re: Fire declared in OVH SBG2 datacentre building

#317

Post by manishsharan »

And this is why the big 3 will continue to dominate. AWS, Microsoft and Google can throw in a lot more money at their phyiscal infrastructure than any other cloud provider.

After this sorry episode, I dont think any CTO or CIO of any public company will be able to even consider using the other guys.

edit: I am not implying that we put all eggs in one basket with no failover and dr. I am implying the big cos will pay 2x premium on infrastructure to project reliability.

Re: Fire declared in OVH SBG2 datacentre building

#318

Post by account42 »

pmontra wrote: post #100

Some pictures of the building with firefighters at work https://www.dna.fr/faits-divers-justice/2021/03/10/strasbour... Edit: Video at https://www.youtube.com/watch?v=a9jL_THG58U Satellite view of the site on Google Maps https://goo.gl/maps/L2T6YNFCtiyDdiNv7

Your Google Maps link seems to be off by a bit.

Re: Fire declared in OVH SBG2 datacentre building

#319

Post by tomxor »

geocrasher wrote: post #10

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2…

Basically the same (offsite backups), but the details are in the what and how which is subjective... For my purposes I decided that offsite backups should only comprise user data and that all server configuration be 100% scripted with some interactive parts to speed up any customization including recovering backups. I also have my own backup servers rather than using a service, and implement immutable incremental backups with rotated ZFS snapshots (this is way simpler than it sounds) - I can highly recommend ZFS as an extremely reliable incremental backup solution but you must enable block level deduplication and expect it to gobble up all the server RAM to be effective (but that's why I dedicate a server to it and don't need masses of cheap slow storage)... also the backup server is restorable by script and only relies on having at least one of the mirrored block devices in tact which I make a local copy of occasionally.

I'm not sure how normal this strategy is outside of container land but I like just using scripts, they are simple and transparent - if you take time and care to write them well.

Re: Fire declared in OVH SBG2 datacentre building

#320

Post by klingon78 »

buran77 wrote: post #231
eb0la wrote: post #200

Earlier quoted context omitted.

Are you truly paranoid? If my money and/or job depended on having something running without (or with minimal) disruption I would be as paranoid as you, too. BTW - Some people call this business recovery plan, not plain paranoia ;-)

It's not paranoia if you're right. All of the risks GP is protecting against are things that happen to someone every day, and they should be seen like wearing the seat belt in a car.

I have a reliability and risk avoidance mindset, but I’ve had to stand back because my mental gas tank for trying to keep things going is near empty.

I’ve really struggled working with others that either are both ignorant and apathetic about the business’s ability to deal with risk or believe that it’s their job to keep putting duct tape over the duct tape that breaks multiple times a day while users struggle.

I like seeing these comments reminding others to a wear seat belt or have backups for their backups, but I don’t know whether I should care more about reliability. I work in an environment that’s a constant figurative fire.

I also like to spend time with my family. I know it’s just a job, and it would be even if I were the only one responsible for it; that doesn’t negate the importance of reliability, but there is a balance.

If you are dedicated to reliability, don’t let this deter you. Some have a full gas tank, which is great.

Post reply on HN