According to the official status page, the whole datacenter is still green http://status.ovh.com/vms/index_sbg2.html
Fire declared in OVH SBG2 datacentre building
271–280 of 613 posts
Re: Fire declared in OVH SBG2 datacentre building
#272According to the official status page, the whole datacenter is still green http://status.ovh.com/vms/index_sbg2.html
It seems to be a static site, which seems reasonable since it aggregates a lot of data and might encounter high load when something goes wrong, so generating it live without caching is not viable. So maybe the server that normally updates it is down too (not that this would be a good excuse)?
10+ hours cache on a status page doesn't look like real-time monitoring to me.
I think this is probably linked to a manual reporting system and they got bigger fish to fry at the moment than updating this status page.
Re: Fire declared in OVH SBG2 datacentre building
#273Earlier quoted context omitted.
If you are a corporate entity of some kind, the final layer of your plan should always be "Go bankrupt". You can't successfully recover from every possible disaster and you shouldn't try to. In the event of a sufficiently unlikely event, your business fails and every penny spent attempting the impossible will be wasted, move on and let professional administrators salvage what they can for your creditors. Lots of peop…
"Go bankrupt" is not a plan. Becoming insolvent might be the end result of a situation but it's not going to help you deal with it. Let's take an example which might lead to bankruptcy. A typical answer to a major disaster (let's say your main and sole building burning as a typical case) for an SME would be to cease activity, furlough employes and stop or defer every payments you can while you claim insurance and ass…
Going bankrupt is a plan. However, it is a somewhat more involved one than it sounds, at first. That's why there should be a corporate lawyer advising on stuff like company structure, liabilities, continuance of pension plans, ordering and reasons for layoffs, etc.
Re: Fire declared in OVH SBG2 datacentre building
#274According to the official status page, the whole datacenter is still green http://status.ovh.com/vms/index_sbg2.html
I feel like there should be place to report infrastructure suppliers with misleading status pages, some kind of crowdsourced database. Without this information, you only find out that they are misleading when something goes very wrong. At best you might be missing out on some SLA refunds, but at worst it could be disasterous for a business. I've been on the wrong side of a update-by-hand status system from a hosting…
Re: Fire declared in OVH SBG2 datacentre building
#275Earlier quoted context omitted.
I'm at OVH as well (in the BHS datacenter, fortunately). I run my entire production system on one beefy machine. The apps and database are replicated to a backup machine hosted with Hetzner (in their Germany datacenter). I also run a tiny VM at OVH which proxies all traffic to Hetzner. I use a failover IP to point at the big rig at OVH. If the main machine fails, I move the failover IP to the VM, which sends all traf…
Are you truly paranoid? If my money and/or job depended on having something running without (or with minimal) disruption I would be as paranoid as you, too. BTW - Some people call this business recovery plan, not plain paranoia ;-)
Enterprise-level projects often have only light protection against wrongful hosting account termination, reasoning that spending a lot of money and having an account manager keeps them safe from clumsy automated systems.
So they might have their primary and replica databases at different DCs from the same hosting provider, and only their nightly backup to a different provider. Four copies to four different providers is a step above three copies with two providers!
A large enterprise would probably be using a filesystem with periodic snapshots, or streaming their redo log to a backup, to protect against a fat-fingered DBA deleting the wrong thing. Of course, filesystem snapshots provide no protection against loss of DC or wrongful hosting account termination, so you might not count them as true backup copies.
Re: Fire declared in OVH SBG2 datacentre building
#276Earlier quoted context omitted.
Easy to see now that a lightly constructed five story cube might not be a fully fire proof.
Would you humor us with a link to a fully fire proof datacenter?
From the outside (and from a position lacking all inside knowledge) it looks highly interconnected and very well ventilated. I'm not sure where you'd put a inert gas supression system or beefy firewalls to slow the fire progress.
Re: Fire declared in OVH SBG2 datacentre building
#277Earlier quoted context omitted.
Once you're big enough to afford a CISO, you're likely big enough to afford office space with decent physical security to serve as a third replicated database site to complement your two datacenters. These solutions are not one-size-fits-all. What works for a small startup isn't appropriate for a 100+ person company.
Yes, I agree. Small companies typically are very bad at security.
Not in my experience. Worked at some small shops that were lightyears ahead in terms of policy, procedures and attitude compared to places I've worked with 50k+ employees globally.
Re: Fire declared in OVH SBG2 datacentre building
#278Earlier quoted context omitted.
> Fire destroyed SBG2 This is crazy. SBG2 was HUGE and if this isn't a translation error on the part of Octave (which I could understand given the stress and ESL) I have a hard time fathoming what kind of fire could destroy a whole facility with nearly 1000 racks of equipment spread out across separated halls. I'm really hoping "destroyed" here means "we lost all power and network core and there's smoke/fire/physical…
This was effectively posted on the outages list as well by someone trustworthy. The pictures also look pretty bad from the outside.
Which outages list? Sounds interesting.
Re: Fire declared in OVH SBG2 datacentre building
#279Earlier quoted context omitted.
"Go bankrupt" is not a plan. Becoming insolvent might be the end result of a situation but it's not going to help you deal with it. Let's take an example which might lead to bankruptcy. A typical answer to a major disaster (let's say your main and sole building burning as a typical case) for an SME would be to cease activity, furlough employes and stop or defer every payments you can while you claim insurance and ass…
Hence the "final layer" statement. Bankruptcy when dealt with correctly is a process not an end. If everything else fail it's better to fill for bankruptcy when there is still something to recover with help of others than to burn everything to ashes because of your vanity. At least that's how I understood parent's comment.
> Bankruptcy when dealt with correctly is a process not an end.
Yes, that's why "Go bankrupt" is not a plan which was the entire point of my reply. That's like saying that your disaster recovery plan is "solve the disaster".
Re: Fire declared in OVH SBG2 datacentre building
#280Earlier quoted context omitted.
If you are a corporate entity of some kind, the final layer of your plan should always be "Go bankrupt". You can't successfully recover from every possible disaster and you shouldn't try to. In the event of a sufficiently unlikely event, your business fails and every penny spent attempting the impossible will be wasted, move on and let professional administrators salvage what they can for your creditors. Lots of peop…
The final layer is call the insurance company.
Not really, the insurance won't make things right in an instant. They will usually compensate you financially, but often only after painstaking evaluation of all circumstances, weighing their chances in court to get out of paying you and maybe a lengthy court battle and a race against your bankruptcy.
So yes, getting insurance can be a good idea to offset some losses you may have, as long as they are somewhat limited compared to your companies overall assets and income. But as soon as the insurance payout matches a significant part of your net worth, the insurance might not save you.