Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OVH outage explained

status.ovh.net

Post reply on HN
Print view View original

81–90 of 129 posts

Re: OVH outage explained

#81

Post by scurvy »

sphix0r wrote: post #7
duncanawoods wrote: post #4

Why are backup generators so unreliable? And that is unreliable by the standard of a non-critical system let alone that of an emergency system where reliability is their sole purpose. Is it just a (non)-survivorship bias where we most commonly talk about the failure cases and they actually have a stellar 99.99999% record that doesn't make headlines?

Similar to many of us make backups in IT but how many actually do try them out and restore?

waves hand

Every day. All of them (automated).

Re: OVH outage explained

#82

Post by toast0 »

vidarh wrote: post #29
bringtheaction wrote: post #17

Earlier quoted context omitted.

Also when you decide on a backup, make sure your backup is not hosting their servers in the same DCs as is your primary, or worse yet that your backup is renting their servers from the same people you rent your primary from.

My lesson in this was not servers, but networks. Back in the mid 90's at some point, Europe largely lost network connectivity to the US for about a day (this is all from recollection; I ran an ISP at the time, and we were affected, but I'm sure I've gotten the occasional detail wrong). Not entirely - there was significant capacity that was technically operational, but in practical terms large subsets of European netw…

This type of outage was common in the US in the 90s as well. Many large (tier 1) ISPs found out that their redundant fiber links were not in redundant bundles, other tier 1s also had important links in the same bundles, and that a backhole had severed the bundle.

Mostly, when this happened, west coast users couldn't contact east coast servers (and vice versa), but occasionally, BGP would work out, and packets would get back and forth, but go around the world.

Re: OVH outage explained

#83

Post by gaadd33 »

joelhaasnoot wrote: post #71
saulrh wrote: post #56

Earlier quoted context omitted.

Hmmm. I didn't actually do the numbers. Let's see... A randomly-googled 2MW diesel generator consumes 50 gallons/hour at quarter load and 160 gallons/hour at full load [1]. So let's say it's 100 gallons/hour to run a 2 MW generator at full load instead of quarter load. The OVH incident report [OP] says that their data center has two cables in each carrying 10 MVA (mega-volts-amperes, ~ watts), giving us 20 MW as roug…

Fuel is much more expensive in Europe. A liter of Diesel is about 1.40, so $6.50 per gallon.

Does Europe tax diesel the same way no matter where it's used? I know in the US diesel for generators is exempt from a lot of taxes resulting in it being significantly cheaper in some states.

Re: OVH outage explained

#84

Post by solatic »

cesarb wrote: post #34

I wonder how easy it is to really test these failover systems. Even if you disconnect the external power (which is risky, since you lose redundancy), it will be a "clean" disconnection, with the power going neatly to zero. I've found a few post-mortems of high-voltage faults, and the waveforms go crazy during the fault, which could lead to failure modes in the transfer switch which wouldn't be found on a "clean" disc…

You'd think that somebody'd start selling some kind of generator that you could hook up the systems to that would generate various forms of unclean power on purpose, precisely so that you could test those rare failure modes.

Re: OVH outage explained

#85

Post by Samon »

cesarb wrote: post #34

I wonder how easy it is to really test these failover systems. Even if you disconnect the external power (which is risky, since you lose redundancy), it will be a "clean" disconnection, with the power going neatly to zero. I've found a few post-mortems of high-voltage faults, and the waveforms go crazy during the fault, which could lead to failure modes in the transfer switch which wouldn't be found on a "clean" disc…

When we had the building UPS replaced several years ago, the contractors accidentally left one of the neutral bars disconnected... everything was up and working fine, the switch was flipped to cut off mains and ensure the backup system could take over, and suddenly there were pops and smoke all over the place as hundreds of desktops, servers, network gear, etc got a dose of ~430v instead of 240. That was a very expensive test.

Re: OVH outage explained

#86

Post by gnu8 »

shrike wrote: post #39

When I was at AWS we were using generators from a large commercial supplier. We were constantly having issues with them refusing to take over if there wasn't sufficient load. Doing so puts lots of stress on a generator and can significantly shorten its life. We went to the manufacturer and tried to get them to make a firmware change; we wanted the generators to sacrifice themselves under most every circumstance (shor…

The company that maintains my generators has a dummy load that is basically a large resistor bank and fan on a trailer. It seems like Amazon could just build something like that on a larger scale to load the generators - short term they could even rent the dummy load trailers and hook them up in the parking lot, right? For a permanent solution, have logic to switch in the appropriate amount of resistance to maintain the minimum load required.

Re: OVH outage explained

#87

Post by Ndymium »

gaadd33 wrote: post #83
joelhaasnoot wrote: post #71

Earlier quoted context omitted.

Fuel is much more expensive in Europe. A liter of Diesel is about 1.40, so $6.50 per gallon.

Does Europe tax diesel the same way no matter where it's used? I know in the US diesel for generators is exempt from a lot of taxes resulting in it being significantly cheaper in some states.

I only know of Finland, but here diesel for work machines is much cheaper than diesel for driving.

Re: OVH outage explained

#88

Post by askvictor »

userbinator wrote: post #70
fulafel wrote: post #66

Earlier quoted context omitted.

Why Diesel generators? Natural gas or gasoline should avoid wet stacking.

Diesel tends to be used more because they tend to be much lower maintenance than other fuel engines, and the cost of the fuel itself is also lower due to the higher efficiency.

Why are they lower maintenance in this situation (i.e. power failsafe generators that are used for only a few hundred hours over their life)? I appreciate that diesel car/truck engines are lower maintenance, but they are used in a very different way.

Re: OVH outage explained

#89

Post by therealmarv »

rmdoss wrote: post #42

I have been a big OVH advocate (cheap prices + free DDoS protection + good hardware) and have been using them for years. However, I am really frustrated by the lack of communication during this incident. All I had was the tweets from the founder, while their status page was down and I had no other place to reach out to them (phone + ticket system down). What is worse is that they treat the individual SBGs (SBG1, SBG2…

Can I ask if vrack itself stayed online with non SBG instances inside? I'm just curious if this also affects their network wide services like vrack or public cloud instances outside SBG.

Re: OVH outage explained

#90

Post by dageshi »

gaadd33 wrote: post #83
joelhaasnoot wrote: post #71

Earlier quoted context omitted.

Fuel is much more expensive in Europe. A liter of Diesel is about 1.40, so $6.50 per gallon.

Does Europe tax diesel the same way no matter where it's used? I know in the US diesel for generators is exempt from a lot of taxes resulting in it being significantly cheaper in some states.

In the UK Diesel is taxed differently for agricultural usage, it typically has a red dye added to it to indicate it's for agricultural use only. I don't honestly know about generators though.
Post reply on HN