Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OVH Incident in Strasbourg

status.ovh.com

Post reply on HN
Print view View original

71–80 of 207 posts

Re: OVH Incident in Strasbourg

#71

Post by jakub_g »

lode wrote: post #4

More info on Twitter from OVH's CEO: https://twitter.com/olesovhcom and on https://twitter.com/ovh_support_en "SBG: ERDF is trying to find out the default. 2 separated 20kV lines are down. We are trying to restart 2 generators A+B for SBG1/SG4. 2 others generators A+B work in SBG2. 1 routing room is in SBG1, the second in SBG2. Both are down. " "An incident is ongoing impacting our network. We are all on the problem.…

BTW this seems to be a better status page than the one submitted to HN (which is 404ing)

http://status.ovh.com/

Re: OVH Incident in Strasbourg

#72

Post by therealmarv »

cjsuk wrote: post #66
z3t4 wrote: post #49

Earlier quoted context omitted.

How do you do fail-over ?

Mirror your data onto another provider continuously (log shipping/rsync), Switch DNS. Ansible works for this stuff as it allows you to can the task of “quick get me a production environment up on Linode!” If you can afford some downtime you don’t need a hot standby just roll out everything into new provider and you’re done. I’ve done this on very large scale environments and small ones and it’s achievable for even sm…

I also do this approach. Get a decent, from your infrastructure independent DNS provider and take care of your Ansible scripts. This way in emergency you do a one liner, have a new production server running and change DNS settings.

Re: OVH Incident in Strasbourg

#73

Post by vim_wannabe »

api wrote: post #67
Hates_ wrote: post #59

Trending on Twitter with the hashtag #OVHGATE https://twitter.com/hashtag/OVHGATE?src=hash

We selected their three data center EU region precisely because they were three separate data centers, so not happy. This is clearly bad design. I think we're now going to have to look into multi-provider options. The only way to be solidly up is to be hosted by more than one company at more than one data center. I've also heard stories of billing nightmares where you get locked out of a cloud provider account, so th…

>locked out of a cloud provider account

I guess this is already a reason by its own. It, among other problems, is what happens when we go from small "local" providers you can actually call to automated global providers that cannot provide immediate support even if they tried.

Re: OVH Incident in Strasbourg

#78

Post by mschuster91 »

therealmarv wrote: post #72
cjsuk wrote: post #66

Earlier quoted context omitted.

Mirror your data onto another provider continuously (log shipping/rsync), Switch DNS. Ansible works for this stuff as it allows you to can the task of “quick get me a production environment up on Linode!” If you can afford some downtime you don’t need a hot standby just roll out everything into new provider and you’re done. I’ve done this on very large scale environments and small ones and it’s achievable for even sm…

I also do this approach. Get a decent, from your infrastructure independent DNS provider and take care of your Ansible scripts. This way in emergency you do a one liner, have a new production server running and change DNS settings.

In addition I'd suggest getting a second domain in a TLD operated by another company in another country than the primary TLD is, and teaching your customers/users that both are valid. This protects you from three things:

1) your DNS provider having issues (even Route53 sometimes has them, https://mwork.io/2017/03/14/aws-route53-dns-outage-impacts-l...)

2) legal issues, when one of your domains gets seized or the provider gets pressured into cancelling, just as has happened with Pirate bay, SciHub and friends, gambling or sites with user generated content that may be illegal or frowned upon in some countries or for the latter to the Nazi site Daily Stormer (although I'm glad for it being down, it's a perfect example what can happen in a very short time frame)

3) (edit, after suggestion below) the entire TLD going down because the TLD DNS provider has issues, which also happens from time to time.

Post reply on HN