Post-Mortem for Google Compute Engine’s Global Outage on April 11 • Hacker News Board

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#131

Post by LaFolle » Wed, Apr 13, 2016, 6:43 PM UTC

nostrademons wrote: post #113

teraflop wrote: post #31

> There are a number of lessons to be learned from this event -- for example, that the safeguard of a progressive rollout can be undone by a system designed to mask partial failures -- ... This is a really important point that should be more generally known. To quote Google's own "Paxos Made Live" paper, from 2007: > In closing we point out a challenge that we faced in testing our system for which we have no systemat…

Google actually does have a systematic solution: fault injection. Google's systems are designed so that you can (manually, if you have the right privileges) tell an RPC to fail regardless of whether it would otherwise have succeeded, and then test the response of the system as a whole. The problem is that these failure cases are exercised much less frequently than the "normal execution" code paths are. For example, e…

Direct link to DiRT => http://queue.acm.org/detail.cfm?id=2371516

I remember my founder (ex Googler) telling us about fault injection at Google. We were pretty amazed by the idea. Thanks for the link @nostrademons.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#132

Post by peterwwillis » Wed, Apr 13, 2016, 6:44 PM UTC

nostrademons wrote: post #113

teraflop wrote: post #31

> There are a number of lessons to be learned from this event -- for example, that the safeguard of a progressive rollout can be undone by a system designed to mask partial failures -- ... This is a really important point that should be more generally known. To quote Google's own "Paxos Made Live" paper, from 2007: > In closing we point out a challenge that we faced in testing our system for which we have no systemat…

Google actually does have a systematic solution: fault injection. Google's systems are designed so that you can (manually, if you have the right privileges) tell an RPC to fail regardless of whether it would otherwise have succeeded, and then test the response of the system as a whole. The problem is that these failure cases are exercised much less frequently than the "normal execution" code paths are. For example, e…

Testing doesn't detect failure, it only detects the failure of a test. Real failures happen more often than test failures, for the same test on the same code with the same input and output. The best systematic solution would detect real failures, not see what happens when you fail a test.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#133

Post by username223 » Wed, Apr 13, 2016, 6:46 PM UTC

duaneb wrote: post #84

brianwawok wrote: post #38

Earlier quoted context omitted.

So crashes like that include drunk drivers, drugged drivers, and texting drivers. What is the crashes per miles for a paying attention driver? If it is 1 per million miles, the self driving car would need to be a lot lower. Now if it was 4am and I am falling asleep at the wheel, I bet any self driving car would beat me. So cool to turn on, but maybe not for a daytime cruise...

Is crashes/mile the best metric or mistakes/mile that could lead to a crash? I certainly make a TON of mistakes that I can correct before they lead to problems--most of them are mild, like having to break a half second later than I'd like, but i bet you are UNDERestimating the improvement due to self driving cars.

It's definitely crashes per mile. Unexpected stuff happens all the time on real roads -- black ice, debris, animals, flat tires -- and safety depends upon the driver's or software's ability to deal with them. It's easy for self-driving not to wander into another lane because they're changing the radio, but it's hard for them to deal with situations their designers didn't anticipate.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#134

Post by godgod » Wed, Apr 13, 2016, 6:46 PM UTC

brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

Imagine the day when the software powering that self driving car chooses to avoid the pedestrian by driving your car off a cliff. Software can kill.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#135

Post by Donzo » Wed, Apr 13, 2016, 6:47 PM UTC

brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

A 1-in-one-million occurrence will happen a thousand times each day when you perform a billion operations.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#136

Post by darawk » Wed, Apr 13, 2016, 6:47 PM UTC

Benjammer wrote: post #126

gliderShip wrote: post #107

Earlier quoted context omitted.

Well No, There is an upper limit on the damage a bad driver can do by say crushing his car with a bus or something like that. Imagine a bug or malware triggered at the same moment world-wide. It could kill millions. So it not as simple as 'It just has to be better than a human'

If a bug can kill millions then it's not "better than a human" though, right?

Depends on how unlikely that bug is :)

probability x value, etc.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#137

Post by nostrademons » Wed, Apr 13, 2016, 6:50 PM UTC

peterwwillis wrote: post #132

nostrademons wrote: post #113

Earlier quoted context omitted.

Google actually does have a systematic solution: fault injection. Google's systems are designed so that you can (manually, if you have the right privileges) tell an RPC to fail regardless of whether it would otherwise have succeeded, and then test the response of the system as a whole. The problem is that these failure cases are exercised much less frequently than the "normal execution" code paths are. For example, e…

Testing doesn't detect failure, it only detects the failure of a test. Real failures happen more often than test failures, for the same test on the same code with the same input and output. The best systematic solution would detect real failures, not see what happens when you fail a test.

That's monitoring, then. As Steve Yegge's Platforms Rant [1] mentioned, testing and monitoring are two sides of the same coin. Google does both, but the original thread-starter here was asking about how to detect failures when the system itself is designed to mask & recover from failures. (FWIW, most such systems do log when they've encountered a failure condition and recovered from it, and this stat is available to the monitoring system.)

[1] https://plus.google.com/+RipRowan/posts/eVeouesvaVX

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#138

Post by brandmeyer » Wed, Apr 13, 2016, 6:53 PM UTC

teraflop wrote: post #31

> There are a number of lessons to be learned from this event -- for example, that the safeguard of a progressive rollout can be undone by a system designed to mask partial failures -- ... This is a really important point that should be more generally known. To quote Google's own "Paxos Made Live" paper, from 2007: > In closing we point out a challenge that we faced in testing our system for which we have no systemat…

The standard solution in realtime safety-critical systems is to perform health monitoring in addition to robust fallbacks, such that when the system is falling back, it is reported as unhealthy.

For example, the CAN bus normally has an automatic retry feature on a variety of errors. A properly functioning CAN bus should have a bit error rate that is nearly zero. Lightly loaded, it can tolerate a very high error rate (say, due to noise, poor termination, etc). In that situation, the product would report a specific warning message to higher-level SCADA systems, such that it gets bubbled up all the way to the operators.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#139

Post by swills » Wed, Apr 13, 2016, 6:54 PM UTC

The thing that stood out for me was:

"...team...worked in shifts overnight..."

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#140

Post by amelius » Wed, Apr 13, 2016, 6:55 PM UTC

ben_jones wrote: post #64

brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

Self driving cars don't have to be perfect. They just have to be safer then driving is today [1]. The real question is if society can handle the unfairness that is death by random software error vs. death by negligent driving. It's easy to blame negligent driving on the driver, we're clearly not negligent so it really doesn't effect us right? But a software error might as well be an act of god, it's something that mi…

> Self driving cars don't have to be perfect. They just have to be safer then driving is today

But how is Google or any other manufacturer going to test their software updates? Are they going to test-drive their cars for tens of thousands of miles over and over again for every little update?