Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OVH Incident in Strasbourg

status.ovh.com

Post reply on HN
Print view View original

161–170 of 207 posts

Re: OVH Incident in Strasbourg

#161

Post by onestone »

dx034 wrote: post #14
qeternity wrote: post #9

Earlier quoted context omitted.

Where did he say GRA was down?

Looks like he deleted the tweet but he wrote earlier that GRA was down with the others while BHS was up. I'd guess he is in RBX and couldn't reach GRA so assumed it was down until they realised that the RBX-GRA line is down. EDIT: He did indeed delete the tweet, [1] is the url in case anyone knows a website archiving them quickly enough. [1] https://twitter.com/olesovhcom/status/928536233311076353

I have several servers in GRA which were unreachable for two hours. Might have been due to the routing or DDoS-protection infrastructure.

Re: OVH Incident in Strasbourg

#162

Post by hinkley »

matthewmacleod wrote: post #94
dx034 wrote: post #46

Earlier quoted context omitted.

So what? Losing main power is a standard case for any DC. That's why you have generators. Even a generator failure is nothing out of the ordinary. But that no generators in a DC work kind of indicates that they don't test them as often as you would expect. They just announced that they want to be a "hypercloud" provider on the scale of AWS and Google Cloud. I really hope that a power failure in Virginia couldn't brin…

It doesn't matter how much testing you do – unfortunately some things can still fail. I'm much more interested in why the failure cascaded to other data centres – that's exactly what shouldn't happen.

Didn’t AWS have a cascading failure last year because of a missing interlock on a script to take part of a data center offline? They discovered that it takes a lot longer than they remembered to reintroduce that many machines to the cluster. They came away from that incident with two or three action items.

Re: OVH Incident in Strasbourg

#164

Post by hinkley »

jbb67 wrote: post #56
dx034 wrote: post #46

Earlier quoted context omitted.

So what? Losing main power is a standard case for any DC. That's why you have generators. Even a generator failure is nothing out of the ordinary. But that no generators in a DC work kind of indicates that they don't test them as often as you would expect. They just announced that they want to be a "hypercloud" provider on the scale of AWS and Google Cloud. I really hope that a power failure in Virginia couldn't brin…

When I've seen things like this before, it's often been the switchover hardware that fails, not the generator as such. it's much harder to test that as you don't want to tell your customers "sorry your sever went down, we were just testing if the switchover worked and it didn't"

I worked for a company doing mostly on-premises (wireless) carrier software years back but they still wanted our own server room to be 3 nines for reasons lost to me now. They had installed new electrical circuits high on the walls so that we could survive minor flooding incidents.

So our lead Ops guy is unplugging half the redundant power supplies and plugging them into the new circuits, but a few critical servers are single PSU still, on UPS units. This is how he discovers that one of our PSUs has rotted and only has about five seconds of reserve power in it. Big outage, no bueno.

Re: OVH Incident in Strasbourg

#165

Post by cjsuk »

hinkley wrote: post #158
cjsuk wrote: post #53

Earlier quoted context omitted.

To be fair people do test generators on a monthly schedule usually. Problem you find is it’s getting colder now so any problems are amplified suddenly. Might have been entirely tested a couple of weeks ago.

Data centers can go down while testing the generators too. There was a DC in California a few years back that had three generators fail during a scheduled test and one of their server rooms had a blackout because of it.

This is a very good point. In all the companies I've worked in we've had more UPS outages than line outages as well.

Re: OVH Incident in Strasbourg

#166

Post by lampington »

nolok wrote: post #152
switch007 wrote: post #142

Earlier quoted context omitted.

> - naaah it's fine, we do not have time nor resources" Yup, been there multiple times in smaller hosting companies. It's basically how it goes. They don't get serious about outages until revenue is severely affected and the brand damaged, they don't get serious about security until there's been a big breach or sales are lost because of lack of certification.

Calling OVH a "smaller hosting company" (which you did do indirectly) is rather funny.

Unless they meant "smaller hosting companies [than OVH]"?

Re: OVH Incident in Strasbourg

#167

Post by dx034 »

zaarn wrote: post #95
hvidgaard wrote: post #62

Earlier quoted context omitted.

If your generators are not reliable in the cold, the issue is the placement of them. This is something basic to account for.

The problem is usually not "not reliable in cold" but rather, the generators are X years old and the temperature is now changing in Europe from "mostly warm" to "warm over the day and icecold in the night" and finally aiming for "icecold all day", which means any equipment exposed will go through rather severe temperature changes. While generators are usually able to handle this with sufficiently low failure risk, th…

It wasn't even freezing overnight at Strasbourg [1], I doubt that the cold could cause any effects at 5C. Maybe they were run once a month but never tested under load? 2x20kv lines failing will have put them at near maximum load immediately, perhaps they weren't designed for that.

[1] https://www.accuweather.com/en/fr/strasbourg/131836/daily-we...

Re: OVH Incident in Strasbourg

#168

Post by dx034 »

jedisct1 wrote: post #163

Not "all datacenters". Only 2 of them. They have 22, not counting all the POPs.

9 by their count (7xRBX+2xSBG). When this was posted, the CEO wrote that everything except BHS (Canadian DC) was offline (tweet now deleted). Presumably he was in RBX and noticed that they couldn't reach any of the other DCs. So it looked like all DCs were down for a while.

Re: OVH Incident in Strasbourg

#169

Post by PuffinBlue »

krallja wrote: post #155
PuffinBlue wrote: post #149

Earlier quoted context omitted.

I'm completely unfamiliar with electrical generators/power generation, so take this question in the spirit of ignorance: Is there not a way to test generators without actually having them power the live datacenter infrastructure? I mean, simulate the exact generation and load requirements that the generators will face? I don't know if it's feasible to dump all that power to ground or whatever, but that way you could…

> dump all the power to ground You know all that cooling equipment data centers tend to have? Dumping power to ground is also known as an electric furnace. Now you have twice as much heat to move, but you only have the usual cooling system. Toasty servers are sad servers. Toasty engineers are dead engineers.

Ok your reply makes no sense to me (see first line of my original comment).

I don't understand why you'd be putting a generator test load through the servers in addition to the normal power supply??? Why would you do that???

I would have thought that you have normal operation going on in the datacentre, normal cooling infrastructure, normal power coming in. Then the generators turned on with their in built cooling, then them delivering power to the test area and not the datacenter - which is then the only additional area you'd need to cool.

Seems like dealing with the heat would be not that hard, but perhaps I have too much faith in engineers? :-)

Even just a massive heating element in a tank of water, giant kettle style would work wouldn't it? Big kettle mind you, but big tank too. Seems like a cheap way to test the generator at full load for an extended test?

Re: OVH Incident in Strasbourg

#170

Post by vultour »

notwedtm wrote: post #102
drchaos wrote: post #98

This affects DNS as well, since domaindiscount24 (a rather large registrar in Germany) happens to host all three of their nameservers with OVH. Just in case you wonder why your sites don't work, even if you host them somewhere else.

This seems like really poor planning on domaindiscount24's part.

This sems lke something I'd expect from someone called domaindiscount24
Post reply on HN