I had never heard of this company til I saw this post. Shrugged, thought, "huh, wonder who that's affecting."
Opened up Age of Empires II....no connection. Go to website for game servers..."Our provider, OVH, is down...."
Go figure.
OVH Incident in Strasbourg
81–90 of 207 posts
Re: OVH Incident in Strasbourg
#82Earlier quoted context omitted.
Huh, I was just looking at them too. Contrary to popular opinion, I kinda prefer it when these things happen before I sign up so that in the post-mortem usually whatever architectural failure lead to the outage is corrected and you get a stronger service. Usually.
Not saying this is the case here but that's also sometimes where you spot amateurism and should run away. I remember a host provider a long time ago (15y) who was storing its backups on the same machine as the main data. Guess how I figured out!
Oh man that sounds bad! You're right, sometimes these events do expose the inability to handle failure and you're right that that means walking away.
Thankfully many times we're reminded that there are good people out there working hard against difficult constraints and they finally get their chance to do things 'correctly' in the wake of the SHTF.
Re: OVH Incident in Strasbourg
#83I had never heard of this company til I saw this post. Shrugged, thought, "huh, wonder who that's affecting." Opened up Age of Empires II....no connection. Go to website for game servers..."Our provider, OVH, is down...." Go figure.
OVH is rather popular in Europe atleast (alongside Hetzner and 1&1).
Re: OVH Incident in Strasbourg
#84Title is misleading. Only RBX and SBG were affected.
06:15 UTC SBG serves failed.
OVH network weathermap: http://weathermap.ovh.net
Btw. First post: https://news.ycombinator.com/item?id=15660524
Re: OVH Incident in Strasbourg
#85I had never heard of this company til I saw this post. Shrugged, thought, "huh, wonder who that's affecting." Opened up Age of Empires II....no connection. Go to website for game servers..."Our provider, OVH, is down...." Go figure.
OVH is a giant hosting company in Europe, providing dedicated servers to quite a lot of companies. Half the internet here was borked. A translation service, a few developer tools, etc...
Sure, it's less dramatic than AWS going down, but it still hits, and hard.
Re: OVH Incident in Strasbourg
#86To make error is human. To propagate error to all server in automatic way is #devops. - @devopsborat
Pfft network engineers have been at it way longer :p
Re: OVH Incident in Strasbourg
#87More info on Twitter from OVH's CEO: https://twitter.com/olesovhcom and on https://twitter.com/ovh_support_en "SBG: ERDF is trying to find out the default. 2 separated 20kV lines are down. We are trying to restart 2 generators A+B for SBG1/SG4. 2 others generators A+B work in SBG2. 1 routing room is in SBG1, the second in SBG2. Both are down. " "An incident is ongoing impacting our network. We are all on the problem.…
BTW this seems to be a better status page than the one submitted to HN (which is 404ing) http://status.ovh.com/
The status page was down during the outage.
Re: OVH Incident in Strasbourg
#88It started with all our SBG servers going down simultaneously. Approximately 1h later all our RBX servers went down as well including the OVH status page and all other OVH web applications. Either their SBG and RBX data centers are somehow connected or those are indeed two independent incidents.
[deleted]
Re: OVH Incident in Strasbourg
#89Earlier quoted context omitted.
I also do this approach. Get a decent, from your infrastructure independent DNS provider and take care of your Ansible scripts. This way in emergency you do a one liner, have a new production server running and change DNS settings.
In addition I'd suggest getting a second domain in a TLD operated by another company in another country than the primary TLD is, and teaching your customers/users that both are valid. This protects you from three things: 1) your DNS provider having issues (even Route53 sometimes has them, https://mwork.io/2017/03/14/aws-route53-dns-outage-impacts-l... ) 2) legal issues, when one of your domains gets seized or the pro…
Good advice. We had some serious trouble when .io went down a few weeks back.
Re: OVH Incident in Strasbourg
#90Earlier quoted context omitted.
We selected their three data center EU region precisely because they were three separate data centers, so not happy. This is clearly bad design. I think we're now going to have to look into multi-provider options. The only way to be solidly up is to be hosted by more than one company at more than one data center. I've also heard stories of billing nightmares where you get locked out of a cloud provider account, so th…
>locked out of a cloud provider account I guess this is already a reason by its own. It, among other problems, is what happens when we go from small "local" providers you can actually call to automated global providers that cannot provide immediate support even if they tried.
That's the problem with this trade off. The small providers tend to have good support and someone you can reach if anything goes wrong but they won't have experts on site 24/7. The large ones have dozens of them on call at all times but lack in support (unless you pay a lot).