Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OVH outage explained

status.ovh.net

Post reply on HN
Print view View original

101–110 of 129 posts

Re: OVH outage explained

#101

Post by kylegordon »

shrike wrote: post #39

When I was at AWS we were using generators from a large commercial supplier. We were constantly having issues with them refusing to take over if there wasn't sufficient load. Doing so puts lots of stress on a generator and can significantly shorten its life. We went to the manufacturer and tried to get them to make a firmware change; we wanted the generators to sacrifice themselves under most every circumstance (shor…

> There is also a great story about an AWS authored firmware update on a set of louver microcontrollers causing a partial outage.

Do tell :-)

Re: OVH outage explained

#102

Post by smoyer »

pbecotte wrote: post #51
shrike wrote: post #39

When I was at AWS we were using generators from a large commercial supplier. We were constantly having issues with them refusing to take over if there wasn't sufficient load. Doing so puts lots of stress on a generator and can significantly shorten its life. We went to the manufacturer and tried to get them to make a firmware change; we wanted the generators to sacrifice themselves under most every circumstance (shor…

Seems like you would have to disable emissions systems to get what you described, which would be illegal ;) The actual solution to that problem, of course, is to use smaller generators?

Or you can fudge your emissions testing.

Re: OVH outage explained

#103

Post by mioelnir »

ngrilly wrote: post #78
snom380 wrote: post #73

Earlier quoted context omitted.

I agree it's a bug, but this is far, far more difficult to test against with a chaos monkey than one might think. With a whole datacenter down, not to speak of multiple availability zones, problems and dependencies will crop up that are extremely difficult to anticipate. Maybe you expected one other AZ to be up so you could provision servers from there, but now you have to somehow get one of them bootstrapped. Or may…

Thanks for the great explanation. I agree that a Chaos Monkey test something a lot more easier to recover from than a whole datacenter getting down. I remember having read somewhere about some company (Facebook/Google/Amazon/Twitter/Dropbox or something at the same scale) that regularly simulates a whole datacenter failure, which made believe it is possible to automatically recover from this. Are you saying that even…

Simulated DC failure is more often then not just traffic flow engineering. It is more about testing the DC that takes over the traffic than it is about testing service restart in the inactive DC.

There is little to test about the introduction of a hard fault, but the service resumption in the other DC is full of data to analyze. Also, in such a setup, getting the fault location running again is not on a hard clock, since it is about restoring redundancy instead of the service.

Re: OVH outage explained

#104

Post by smoyer »

cperciva wrote: post #65
gedrap wrote: post #37

Earlier quoted context omitted.

That applies to other kinds of reliability testing too, especially everything that goes through a network. It's easy to handle when something is completely unavailable (i.e. instant error), but when something, be it a database or some endpoint, is available but horribly slow, that's a whole different thing.

Seems like we need systems designed with "suicide" mechanisms built in, so that if they detect that they have a poor quality of life (err, I mean, that they're providing a poor quality of service) they'll shut down completely.

The circuit-breaker pattern is commonly used to make the affected system appear off-line to it's upstream dependents - you don't need to actually shut the server down completely and will probably want to log into it to see what's happening. If the dependent system can't function without the affected system, it should also indicate an error and further upstream systems should detect it.

In your house, you have the circuit-breaker pattern implemented in hardware. And upstream of your house there are many more layers of circuit breakers that generally increase in size until they reach a point where there is redundancy. A circuit-breaker going off in your house protects the other circuits in your house, a circuit-breaker going off on your street protects the rest of your neighborhood.

Industrial circuit-breakers are commonly a combination of hardware and software. Personally, I've lost more equipment due to brown-outs than any other cause. If you have equipment you don't want to experience a brown-out, program the breaker to cut off power completely.

Re: OVH outage explained

#105

Post by tete »

drdebug wrote: post #27
wereHamster wrote: post #24

Earlier quoted context omitted.

I would be pissed if my hosting provider randomly restarted my servers, taking them offline for minutes each time.

So would I, but that actually makes me realize that I should be doing more of what netflix is doing too and ensure I can randomly restart servers without affecting service.

Isn't choasmonkey just randomly restarting virtual servers on Amazon?

Kudos to Netflix, but restarting a virtual server vs a physical or a whole data center are different things.

I think every company that cares a bit about high availability knocks out stuff randomly or at least in different ways, does stuff like introducing packet loss, etc. It's another layer and another thing to test that on service/virtual server layer than on close to physical layers.

Of course, one should test that too and it's nowhere near impossible, but Chaosmonkey is for a somewhat different use case.

Also the "article" mentions that tests are done.

Re: OVH outage explained

#106

Post by yourapostasy »

shrike wrote: post #39

When I was at AWS we were using generators from a large commercial supplier. We were constantly having issues with them refusing to take over if there wasn't sufficient load. Doing so puts lots of stress on a generator and can significantly shorten its life. We went to the manufacturer and tried to get them to make a firmware change; we wanted the generators to sacrifice themselves under most every circumstance (shor…

Apart from the other ideas for picking up load, if you have land for it, then there are many productive ways to use excess generation capacity that can still be reliable, especially if a conventional resistive load is available on standby. Some methods can even bolster the DC’s own opex. None are cheaper than a resistive load, but I wonder if the resistive load was modeled as requiring high availability, then what is the pricing of the excess capacity?

* Plasma arc garbage incineration. Not smelly if the input arrives in a ceramic sealed container.

* Glass cullet smelter producing glass foam insulation bricks. Turn excess power into additional DC modular insulation. My personal favorite, because this is a giant resistive load that directly supports the DC’s bottom line.

* Aluminum recycling smelter. Build additional heat sinks for increasing rack efficiency.

* Distilled water generation. Route it back through water chillers, cut down on mineral scaling damage over time.

Re: OVH outage explained

#107

Post by lima »

yourapostasy wrote: post #106
shrike wrote: post #39

When I was at AWS we were using generators from a large commercial supplier. We were constantly having issues with them refusing to take over if there wasn't sufficient load. Doing so puts lots of stress on a generator and can significantly shorten its life. We went to the manufacturer and tried to get them to make a firmware change; we wanted the generators to sacrifice themselves under most every circumstance (shor…

Apart from the other ideas for picking up load, if you have land for it, then there are many productive ways to use excess generation capacity that can still be reliable, especially if a conventional resistive load is available on standby. Some methods can even bolster the DC’s own opex. None are cheaper than a resistive load, but I wonder if the resistive load was modeled as requiring high availability, then what is…

The generator is not going to run unless there's an outage.

Re: OVH outage explained

#108

Post by bpicolo »

shrike wrote: post #39

When I was at AWS we were using generators from a large commercial supplier. We were constantly having issues with them refusing to take over if there wasn't sufficient load. Doing so puts lots of stress on a generator and can significantly shorten its life. We went to the manufacturer and tried to get them to make a firmware change; we wanted the generators to sacrifice themselves under most every circumstance (shor…

> The manufacturer refused

That's how you create business opportunities for others, heh

Re: OVH outage explained

#109

Post by msh »

sphix0r wrote: post #7
duncanawoods wrote: post #4

Why are backup generators so unreliable? And that is unreliable by the standard of a non-critical system let alone that of an emergency system where reliability is their sole purpose. Is it just a (non)-survivorship bias where we most commonly talk about the failure cases and they actually have a stellar 99.99999% record that doesn't make headlines?

Similar to many of us make backups in IT but how many actually do try them out and restore?

We have yearly recovery exercises for randomly chosen subsets of servers and databases.

Re: OVH outage explained

#110

Post by robryk »

Johnny555 wrote: post #67
robryk wrote: post #23

Earlier quoted context omitted.

The particular failure mode you're talking about is reasonably easy to reliably remove: all fuel piping should be under lower than atmospheric pressure. You accomplish this by installing suction pumps next to consumers (and install additional pumps if the height difference is too large). That way, if a pipe breaks (except for the short pieces of pipe between the pipe and consumer), there will be no sustained fuel lea…

Building constraints don't always make it that easy -- the generator could be below the tank level, so siphon action can still siphon the tank empty, or it can be a few floors about the tank, too high to suck fuel from the tank so it has to be pushed up from below.

> the generator could be below the tank level

This is what anti-siphon valves are used to prevent: they are valves that open only when sufficient suction is present in the fuel line leaving the tank. Thus, when the line breaks or the pump fails, the valve will close.

> or it can be a few floors about the tank, too high to suck fuel from the tank so it has to be pushed up from below.

True, it has to be pushed up then. Alas, you can have a sequence of suction pumps. Note that otherwise you'd need to have high-pressure fuel piping, which might cause more problems than the possibility of feeding a fire indefinitely.

Post reply on HN