Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OVH Incident in Strasbourg

status.ovh.com

Post reply on HN
Print view View original

141–150 of 207 posts

Re: OVH Incident in Strasbourg

#141

Post by fapjacks »

therealmarv wrote: post #21

wow, yesterday I was playing with their public cloud because considering choosing them. I had some connection problem with my private networking there (deleted it more than once) and opened a ticket. If it was me... sorry, haha. Not good advertisement but it can happen to everyone.

Hey I would suggest not completely disregarding OVH. Their prices are good and their network is usually very reliable and fast. You simply won't find a North American hosting provider with those prices and reliability. You won't find one at double OVH prices, either (and I have tried).

Re: OVH Incident in Strasbourg

#142

Post by switch007 »

KeitIG wrote: post #18

I imagine Mr Good Guy at OVH telling some others: "guys we have a single point of failure in our architecture with SBG, maybe we should... - naaah it's fine, we do not have time nor resources" Then shit happens. edit: I have no idea what is happening exactly, but OVH being what it is, it seems extremely weird that all datacenters "can" get down at the same time, and it looks like a serious architecture problem to me…

> - naaah it's fine, we do not have time nor resources"

Yup, been there multiple times in smaller hosting companies.

It's basically how it goes. They don't get serious about outages until revenue is severely affected and the brand damaged, they don't get serious about security until there's been a big breach or sales are lost because of lack of certification.

Re: OVH Incident in Strasbourg

#143

Post by dx034 »

vabene1111 wrote: post #123
tyingq wrote: post #109

Earlier quoted context omitted.

>ts OVH: The Hardware is good DDoS protection is good The Prices are high The prices are high? Compared to what? Cheap is their raison d'être.

compared to other dedicated Server Hardware, not talking about business Cloud Infrastructure, no idea about that. Sry if that caused confusion

Who's cheaper on the dedicated side? Hetzner can be a bit cheaper than soyoustart but wouldn't be aware of anyone else (with reasonable quality).

Re: OVH Incident in Strasbourg

#144

Post by zaarn »

dx034 wrote: post #143
vabene1111 wrote: post #123

Earlier quoted context omitted.

compared to other dedicated Server Hardware, not talking about business Cloud Infrastructure, no idea about that. Sry if that caused confusion

Who's cheaper on the dedicated side? Hetzner can be a bit cheaper than soyoustart but wouldn't be aware of anyone else (with reasonable quality).

I've tried Hetzner but the Network Peering to Telekom is subpar compared to OVH. On Hetz I got about 40Mbps up/down to my local computer while on OVH I can easily load my DSL 100% without issues.

Re: OVH Incident in Strasbourg

#145

Post by dx034 »

They now posted their explanation [1] but I don't buy it. I find it hard to believe that the RBX incident happened shortly after the SGB incident without any connection between these two. They should have redundant networking (at least that's what they say) so one corrupted DB in RBX shouldn't have brought down the whole DC (or 7 DCs according to their system). Maybe they pulled corrupt data from SGB because it was down but I don't believe that at the same time of a power failure, two redundant network nodes got corrupted without any notice. Otherwise wouldn't that mean that one hardware issue can also bring down a whole region?

[1] http://status.ovh.net/?do=details&id=15162&PHPSESSID=7220be2...

Re: OVH Incident in Strasbourg

#146

Post by zaarn »

pfg wrote: post #8
dx034 wrote: post #2

Some servers in GRA still appear to work if that's of any help. All data centres offline at once sounds more like an attack than a power failure in one location. According to them, there was a power failure in SBG but I don't see how that should affect routing in data centres several hundred miles away. https://twitter.com/olesovhcom/status/928541667283623936 EDIT: Maybe related to the Cisco issue? https://blogs.cisc…

It seems more likely that their data centers aren't quite as isolated as they thought they'd be. The outage also appears to be limited to their locations in Europe.

The two issues are isolated.

One datacenter suffered massive power failure while the other hand network equipment fail at roughly the same time.

Re: OVH Incident in Strasbourg

#147

Post by dx034 »

zaarn wrote: post #144
dx034 wrote: post #143

Earlier quoted context omitted.

Who's cheaper on the dedicated side? Hetzner can be a bit cheaper than soyoustart but wouldn't be aware of anyone else (with reasonable quality).

I've tried Hetzner but the Network Peering to Telekom is subpar compared to OVH. On Hetz I got about 40Mbps up/down to my local computer while on OVH I can easily load my DSL 100% without issues.

OVH is one of the few with direct peering to Telekom, most don't want to pay for that. Otherwise Hetzner is good, their own network is limited but peering works reasonably well. But apart from them I'm not aware of anyone with cheaper dedicated prices than soyoustart

Re: OVH Incident in Strasbourg

#148

Post by teapot01 »

cjsuk wrote: post #53
hvidgaard wrote: post #48

Earlier quoted context omitted.

What is the point of backup generators if you do not verify that they work every so often? I have a very hard time believing that they actually tested that they worked, because a failure of not one, but both of them.

To be fair people do test generators on a monthly schedule usually. Problem you find is it’s getting colder now so any problems are amplified suddenly. Might have been entirely tested a couple of weeks ago.

Generators in critical situations like this are usually provided with crankcase heaters which keeps the oil at temperature and makes them easy to start. Also, any emergency power system should be (and usually is) tested monthly if only to prevent the fuel lines going bad.

Re: OVH Incident in Strasbourg

#149

Post by PuffinBlue »

_wmd wrote: post #136
gyaru wrote: post #105

Earlier quoted context omitted.

People not actually testing emergency equipment.

While at $bigco we halted testing of generation equipment because it was sending DCs offline more often than it kept them up. Lawyers were involved, things got ugly

I'm completely unfamiliar with electrical generators/power generation, so take this question in the spirit of ignorance:

Is there not a way to test generators without actually having them power the live datacenter infrastructure? I mean, simulate the exact generation and load requirements that the generators will face?

I don't know if it's feasible to dump all that power to ground or whatever, but that way you could test the generator under full load at will and identify issues without impacting the (operationally) live datacenter itself.

This equates a bit in my head with the 'verifying the backup' bit you might get in software, whereas actually using the generator to power the live operational datacenter equipment would be more like 'restore from backup'.

I don't know if it's possible though.

Re: OVH Incident in Strasbourg

#150

Post by pfg »

zaarn wrote: post #146
pfg wrote: post #8

Earlier quoted context omitted.

It seems more likely that their data centers aren't quite as isolated as they thought they'd be. The outage also appears to be limited to their locations in Europe.

The two issues are isolated. One datacenter suffered massive power failure while the other hand network equipment fail at roughly the same time.

The network equipment failure was due to a software error[1]. I would not rule out some kind of cascading failure triggered by the initial outage until the cause of the bug is known, though OVH seems comfortable with such a statement.

[1]: http://travaux.ovh.net/?do=details&id=28256

Post reply on HN