Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OVH outage explained

status.ovh.net

Post reply on HN
Print view View original

111–120 of 129 posts

Re: OVH outage explained

#111

Post by robryk »

robocat wrote: post #79
robryk wrote: post #23

Earlier quoted context omitted.

The particular failure mode you're talking about is reasonably easy to reliably remove: all fuel piping should be under lower than atmospheric pressure. You accomplish this by installing suction pumps next to consumers (and install additional pumps if the height difference is too large). That way, if a pipe breaks (except for the short pieces of pipe between the pipe and consumer), there will be no sustained fuel lea…

So now the system fails due to frothy diesel. Or due to failures of the additional pumps (adding mechanical complexity to a system often decreases overall reliability)

Yes. This is a tradeoff between a chance of catastrophic failure (fire fed by fuel from the storage tank) and "normal" failure.

Re: OVH outage explained

#112

Post by yourapostasy »

lima wrote: post #107
yourapostasy wrote: post #106

Earlier quoted context omitted.

Apart from the other ideas for picking up load, if you have land for it, then there are many productive ways to use excess generation capacity that can still be reliable, especially if a conventional resistive load is available on standby. Some methods can even bolster the DC’s own opex. None are cheaper than a resistive load, but I wonder if the resistive load was modeled as requiring high availability, then what is…

The generator is not going to run unless there's an outage.

I forgot to mention that you only need to set up for very small-scale production, almost maker-scale, so not a whole lot of land is required. With automation, any outage will create a steady trickle of usable goods. The DC's I work with run generators once a month to test (and generally, most engines don't like sitting still all the time). Certainly sufficient for use by nearby residents, so co-generation would benefit third parties if not the DC itself.

Re: OVH outage explained

#113

Post by scurvy »

darklajid wrote: post #99
scurvy wrote: post #81

Earlier quoted context omitted.

waves hand Every day. All of them (automated).

Can you turn/expand that into a somewhat longer answer, a list of links that you perused to build that or .. a post on its own?

Brains and know-how my good man, not copypasta.

Re: OVH outage explained

#114

Post by exikyut »

jotm wrote: post #57
exikyut wrote: post #28

Earlier quoted context omitted.

Hmm. Makes me wonder. If generators really are just a giant blob of metal and very simple (and I have no reason to disbelieve the GP comment)... well... it could be kind of interesting to build an open source software stack to handle switchover. Because, disclaimers notwithstanding, the code would ostensibly be super simple too. So even if it couldn't officially be used directly, it would certainly provide a good bas…

The actual diesel engines and generators are simple metal. Not the switching circuits. If the electric controllers can't handle the load/switching speed/whatever, no clever software and/or reliability in the actual generators will help.

Ah.

I've seen what switching 20kV looks like in videos - yeah, that kind of thing requires very careful design, and is invariably going to come entangled with a PLC-style controller, as is the norm for industrial equipment.

Re: OVH outage explained

#115

Post by JoeMalt »

dageshi wrote: post #90
gaadd33 wrote: post #83

Earlier quoted context omitted.

Does Europe tax diesel the same way no matter where it's used? I know in the US diesel for generators is exempt from a lot of taxes resulting in it being significantly cheaper in some states.

In the UK Diesel is taxed differently for agricultural usage, it typically has a red dye added to it to indicate it's for agricultural use only. I don't honestly know about generators though.

You can use red diesel in a generator, I think the legality comes down to whether it's being used to power a vehicle on a public road.

Re: OVH outage explained

#116

Post by tgtweak »

therealmarv wrote: post #89
rmdoss wrote: post #42

I have been a big OVH advocate (cheap prices + free DDoS protection + good hardware) and have been using them for years. However, I am really frustrated by the lack of communication during this incident. All I had was the tweets from the founder, while their status page was down and I had no other place to reach out to them (phone + ticket system down). What is worse is that they treat the individual SBGs (SBG1, SBG2…

Can I ask if vrack itself stayed online with non SBG instances inside? I'm just curious if this also affects their network wide services like vrack or public cloud instances outside SBG.

If power to network gear was down, then it probably affected vrack connectivity, likely on the entirety of sbg.

Re: OVH outage explained

#117

Post by tgtweak »

gnu8 wrote: post #86
shrike wrote: post #39

When I was at AWS we were using generators from a large commercial supplier. We were constantly having issues with them refusing to take over if there wasn't sufficient load. Doing so puts lots of stress on a generator and can significantly shorten its life. We went to the manufacturer and tried to get them to make a firmware change; we wanted the generators to sacrifice themselves under most every circumstance (shor…

The company that maintains my generators has a dummy load that is basically a large resistor bank and fan on a trailer. It seems like Amazon could just build something like that on a larger scale to load the generators - short term they could even rent the dummy load trailers and hook them up in the parking lot, right? For a permanent solution, have logic to switch in the appropriate amount of resistance to maintain…

There are, unfortunately, a lot of things between on and off when it comes to utility power. Over voltage, voltage drop (which can cause fuses to blow since amperage goes up), phase outage, phase imbalance, harmonic noise (causing ground/neutral feedback), micro outages (few cycles; Most downtime I've seen in datacenters has been due to irregular power, not power loss. Case in point.

Dummy loads are good for testing but they are not typically variable - if you had 2.5mva generators and a 1mw dummy load you wouldn't be able to run more than 1mw of critical IT load.

I'll say this is the first I've heard of not having enough load to start a generator. They will happily start up and idle.

Re: OVH outage explained

#118

Post by Feld0 »

ngrilly wrote: post #59
ngrilly wrote: post #19

I have a hard time understanding why they needed humans babysitting the restart of servers and services. Servers and services are supposed to automatically restart after the power came back. This implies OVH doesn't regularly test server restarts with something like Netflix Chaos Monkey. Moreover, the power lines were not really redundant in SBG, and regarding the network downtime in RBX, it looks like the network co…

Some of you are commenting that asking OVH to randomly restart customer servers would be silly. I agree. My comment was about servers managed by OVH. The post-mortem specifically says: "Since then, we have been working on restarting services. Powering the site with energy allows the servers to be restarted, but the services running on the servers still need to be restarted. That's why each service has been coming bac…

OVH monitors customers' servers and investigates if they stop responding to pings, unless you opt out of this. I'm guessing they were looking at this system to make sure all their customers' machines came online again after the incident.

Source: am an OVH customer

Re: OVH outage explained

#119

Post by ngrilly »

Feld0 wrote: post #118
ngrilly wrote: post #59

Earlier quoted context omitted.

Some of you are commenting that asking OVH to randomly restart customer servers would be silly. I agree. My comment was about servers managed by OVH. The post-mortem specifically says: "Since then, we have been working on restarting services. Powering the site with energy allows the servers to be restarted, but the services running on the servers still need to be restarted. That's why each service has been coming bac…

OVH monitors customers' servers and investigates if they stop responding to pings, unless you opt out of this. I'm guessing they were looking at this system to make sure all their customers' machines came online again after the incident. Source: am an OVH customer

Yes, I'm aware of this monitoring service. Is OVH able to login in your server to investigate what is the issue?

Re: OVH outage explained

#120

Post by ngrilly »

cesarb wrote: post #94
ngrilly wrote: post #78

Earlier quoted context omitted.

Thanks for the great explanation. I agree that a Chaos Monkey test something a lot more easier to recover from than a whole datacenter getting down. I remember having read somewhere about some company (Facebook/Google/Amazon/Twitter/Dropbox or something at the same scale) that regularly simulates a whole datacenter failure, which made believe it is possible to automatically recover from this. Are you saying that even…

> I remember having read somewhere about some company (Facebook/Google/Amazon/Twitter/Dropbox or something at the same scale) that regularly simulates a whole datacenter failure It was Google: http://queue.acm.org/detail.cfm?id=2371516 ("Weathering the Unexpected" [2012])

Thanks! This is the article I had in mind.
Post reply on HN