Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-Mortem for Google Compute Engine’s Global Outage on April 11

status.cloud.google.com

Post reply on HN
Print view View original

111–120 of 368 posts

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#111

Post by dward »

ikeboy wrote: post #44
mjibson wrote: post #27

Earlier quoted context omitted.

I'm attempting to even imagine how one would build a useful way to test this. Would they have to have a secondary, world-wide datacenter network with all their various services behind it?

You could have it send messages to the actual servers, but with an added flag that says "fake", which makes the servers ignore the message/send back a message saying pass/fail/whatever (testing the flag could happen first, one server at a time manually). Then check whether the program continued to push updates.

You mean an ICMP request? The IPs were anycast and did not become unreachable until all edge routers had stopped announcing BGP routes. At that point the failure was global. Check out the postmortem, it's a good read.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#112

Post by mikecb »

icebraining wrote: post #35
obulpathi wrote: post #14

> Finally, to underscore how seriously we are taking this event, we are offering GCE and VPN service credits to all impacted GCP applications equal to (respectively) 10% and 25% of their monthly charges for GCE and VPN. These credits exceed what is promised by Google Cloud in their SLA's for Compute Engine and VPN service!

Only barely. They're down to 2.5 minutes of downtime left for the next 30 days if they want to keep the 99.95% level.

They do SLO by quarter.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#113

Post by nostrademons »

teraflop wrote: post #31

> There are a number of lessons to be learned from this event -- for example, that the safeguard of a progressive rollout can be undone by a system designed to mask partial failures -- ... This is a really important point that should be more generally known. To quote Google's own "Paxos Made Live" paper, from 2007: > In closing we point out a challenge that we faced in testing our system for which we have no systemat…

Google actually does have a systematic solution: fault injection. Google's systems are designed so that you can (manually, if you have the right privileges) tell an RPC to fail regardless of whether it would otherwise have succeeded, and then test the response of the system as a whole.

The problem is that these failure cases are exercised much less frequently than the "normal execution" code paths are. For example, every year Google does DiRT [1] exercises which test system responses to a large calamity, eg. a California earthquake that kills everyone in Mountain View and SF including the senior leadership, and also knocks out all west coast datacenters. The half-life of code at Google (in my observation) is roughly 1 year, which means that half of all code has never gone through a DiRT exercise. The same applies to other, less serious fault injection mechanisms: they may get executed once every year or two, and serious bugs can crop up in the meantime. Automated testing of fault injection isn't really feasible, because the number of potential faults grows combinatorially with the number of independent RPCs in the system.

I'd be willing to bet that the two bugs that caused this outage were less than 6 months old. In my tenure at Google, the vast majority of bugs that showed up in postmortems were introduced [1] http://everythingsysadmin.com/2012/09/devops-google-reveals-...

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#114

Post by durandal1 »

duaneb wrote: post #70
infinotize wrote: post #50

Earlier quoted context omitted.

Aircraft autopilots also rely on experienced and licensed pilots to operate them and be responsible for the aircraft at all times. Self driving cars have assume the operator is not particularly capable nor paying attention to anything happening on the road.

Do they rely on the pilot? I was under the impression it was entirely hands off.

There are many conditions unders which an aircraft autopilot will simply disconnect without previous warning and hand over controls to the pilot.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#115

Post by obulpathi »

duskwuff wrote: post #23
obulpathi wrote: post #14

> Finally, to underscore how seriously we are taking this event, we are offering GCE and VPN service credits to all impacted GCP applications equal to (respectively) 10% and 25% of their monthly charges for GCE and VPN. These credits exceed what is promised by Google Cloud in their SLA's for Compute Engine and VPN service!

... which is precisely (almost word-for-word) what the post-mortem goes on to say. Is there something specific you're trying to call attention to here?

Nop. Probably did too much copy-pasting :( Mearly wanted to highlight the point.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#116

Post by mattdeboard »

manquer wrote: post #72
mjibson wrote: post #27

Earlier quoted context omitted.

I'm attempting to even imagine how one would build a useful way to test this. Would they have to have a secondary, world-wide datacenter network with all their various services behind it?

While testing would have been quite difficult, any simple canary release or timed release mechanism would have prevented this / limited the damage. At such mission critical systems, applying any global change in a such manner is asking for it, Devops can also be SPOF, this seems one such case.

If you read the actual report, it mentions that they did a canary step but its effectiveness was undermined.

> In this event, the canary step correctly identified that the new configuration was unsafe. Crucially however, a second software bug in the management software did not propagate the canary step’s conclusion back to the push process, and thus the push system concluded that the new configuration was valid and began its progressive rollout.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#117

Post by lallysingh »

deegles wrote: post #106
xanderjanz wrote: post #96

Earlier quoted context omitted.

I imagine we'd treat it like we do getting hit by a drunk driver. Vilify reckless programmers, and don't think about it.

This would have a dramatic chilling effect on hiring for self-driving car software developers, which would ironically make them less safe.

It might lead to some decent work on formal verification of programs.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#118

Post by pjlegato »

wyldfire wrote: post #105
pjlegato wrote: post #45

Attention startups: this is what incident post-mortems should look like.

Many startups probably don't have anywhere near the same level of SLA nor revenue of GCE.

Of course. You should still have a goal to strive towards.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#119

Post by eslaught »

ben_jones wrote: post #64
brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

Self driving cars don't have to be perfect. They just have to be safer then driving is today [1]. The real question is if society can handle the unfairness that is death by random software error vs. death by negligent driving. It's easy to blame negligent driving on the driver, we're clearly not negligent so it really doesn't effect us right? But a software error might as well be an act of god, it's something that mi…

That assumes that failures are uncorrelated. My personal concern is with correlated failures, like those that occurred in GCE. What if cars from some manufacturer all fail simultaneously in the same way (say, because of a software push that rolled out more aggressively than it just have, just as in the GCE case)? That's the sort of scenario I'm really concerned about.

At least with human drivers, the failures are generally uncorrelated.

Post reply on HN