Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-Mortem for Google Compute Engine’s Global Outage on April 11

status.cloud.google.com

Post reply on HN
Print view View original

121–130 of 368 posts

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#121

Post by mgw »

manquer wrote: post #72
mjibson wrote: post #27

Earlier quoted context omitted.

I'm attempting to even imagine how one would build a useful way to test this. Would they have to have a secondary, world-wide datacenter network with all their various services behind it?

While testing would have been quite difficult, any simple canary release or timed release mechanism would have prevented this / limited the damage. At such mission critical systems, applying any global change in a such manner is asking for it, Devops can also be SPOF, this seems one such case.

They had a canary release mechanism in place. This is described in the post mortem.

> These safeguards include a canary step where the configuration is deployed at a single site and that site is verified to still be working correctly, and a progressive rollout which makes changes to only a fraction of sites at a time, so that a novel failure can be caught at an early stage before it becomes widespread. In this event, the canary step correctly identified that the new configuration was unsafe. Crucially however, a second software bug in the management software did not propagate the canary step’s conclusion back to the push process, and thus the push system concluded that the new configuration was valid and began its progressive rollout.

Taking no cofirmation of the canary testing process as a signal to go ahead though is not just a bug but a design flaw IMO.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#122

Post by windwake12 »

hsod wrote: post #85

> Crucially however, a second software bug in the management software did not propagate the canary step’s conclusion back to the push process, and thus the push system concluded that the new configuration was valid and began its progressive rollout. Perhaps the progressive rollout should wait for an affirmative conclusion instead of assuming no news is good news? I'm not being snarky, there may be some reason they do…

Presumably it received a false positive (or it was interpreted as such). This really seems like the root cause, and I suspect a case of happy path engineering striking again.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#123

Post by pjlegato »

deegles wrote: post #106
xanderjanz wrote: post #96

Earlier quoted context omitted.

I imagine we'd treat it like we do getting hit by a drunk driver. Vilify reckless programmers, and don't think about it.

This would have a dramatic chilling effect on hiring for self-driving car software developers, which would ironically make them less safe.

For this reason, most self-driving car development will only happen at large companies like Google, Tesla, Ford, etc. because they are the only ones who will be able to afford to purchase a massive general liability insurance policy.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#125

Post by ikeboy »

dward wrote: post #111
ikeboy wrote: post #44

Earlier quoted context omitted.

You could have it send messages to the actual servers, but with an added flag that says "fake", which makes the servers ignore the message/send back a message saying pass/fail/whatever (testing the flag could happen first, one server at a time manually). Then check whether the program continued to push updates.

You mean an ICMP request? The IPs were anycast and did not become unreachable until all edge routers had stopped announcing BGP routes. At that point the failure was global. Check out the postmortem, it's a good read.

The servers became unreachable, but the IPs weren't unreachable until all the servers were reconfigured.

My test should have caught this bug:

> In this event, the canary step correctly identified that the new configuration was unsafe. Crucially however, a second software bug in the management software did not propagate the canary step’s conclusion back to the push process, and thus the push system concluded that the new configuration was valid and began its progressive rollout.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#126

Post by Benjammer »

gliderShip wrote: post #107
ben_jones wrote: post #64

Earlier quoted context omitted.

Self driving cars don't have to be perfect. They just have to be safer then driving is today [1]. The real question is if society can handle the unfairness that is death by random software error vs. death by negligent driving. It's easy to blame negligent driving on the driver, we're clearly not negligent so it really doesn't effect us right? But a software error might as well be an act of god, it's something that mi…

Well No, There is an upper limit on the damage a bad driver can do by say crushing his car with a bus or something like that. Imagine a bug or malware triggered at the same moment world-wide. It could kill millions. So it not as simple as 'It just has to be better than a human'

If a bug can kill millions then it's not "better than a human" though, right?

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#127

Post by ceejayoz »

duaneb wrote: post #79
ceejayoz wrote: post #60

Earlier quoted context omitted.

> Yeah, remember, auto-pilot in a plane needs to be 100% reliable, or everyone dies. First, they're not anywhere near 100% reliable. They can fail on their own, and they'll also intentionally shut themselves off if the instruments they rely on fail. https://en.wikipedia.org/wiki/Air_France_Flight_447 Second, an autopilot failure shouldn't lead to death if the pilots are competent and paying attention.

What does a failed autopilot look like? Would a pilot do any better with an "aerodynamic stall"? I know little about planes and it seems like that'd be a big problem with or without a pilot driving.

All pilots are trained to recover from a stall.

A failed autopilot could look like all sorts of things, from just automatically disconnecting itself (usually with a loud warning alert) to issuing incorrect instructions (which is why the pilots are supposed to be awake and alert while it's engaged, watching the instruments).

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#128

Post by SubuSS »

ben_jones wrote: post #64
brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

Self driving cars don't have to be perfect. They just have to be safer then driving is today [1]. The real question is if society can handle the unfairness that is death by random software error vs. death by negligent driving. It's easy to blame negligent driving on the driver, we're clearly not negligent so it really doesn't effect us right? But a software error might as well be an act of god, it's something that mi…

I think the bigger worry is not the perfection part: It is the uniformity part. Considering software will be replicated (ignoring the ML ways of driving for now), updated and refreshed en masse, the impact is going to be very severe. A single nut case shoots up one school or his neighbors. The whole world turning into nut cases is going to be a walking dead scenario.

We see this ALL the time with ALL the big companies including the ones I have worked for in the past. I am very interested in possible solutions people are cooking up here.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#129

Post by nostrademons »

specialist wrote: post #81
cjbprime wrote: post #17

It looks like there were at least three catastrophic bugs present: 1. Evaluated a configuration change before the change had finished syncing across all configuration files, resulting in rejecting the change. 2. So it tried to reject the change, but actually just deleted everything instead. 3. Something was supposed to catch changes that break everything, and it detected that everything was broken but its attempt to…

Spit balling here... For progressive rollouts, what if config changes where pulled instead of pushed? Each system would be responsible for itself updating, verifying (canary, smoketest, make sure other systems successfully updated, etc), bouncing, and then rolling back as needed.

A bunch of that's in-place already, eg. all Google servers have health checks that run basic smoke tests on a configuration, and if a large number of replicas become unhealthy after a config change, the rollout process automatically aborts and rolls back to the last known good conversion.

The problem here was that there was a bug in the health check that masked the problem by assigning the last-good configuration, and then there was a bug in that code that had saved "nothing" as the last-good configuration. So rather than failing and having the error caught at the top level, it failed and buggy failure-recovery code made the problem worse.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#130

Post by marcosdumay »

pveierland wrote: post #74
brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

It also showcases the great thing about self-driving cars. Even though accidents will happen, when it does there will be plenty of sensor data and logs which can be examined to find the exact cause in a post-mortem. An improvement to the software can then be made, and millions of cars deployed can all effectively learn from a single accident. With humans, the amount of knowledge gained and the collective improvement…

Just like planes.

I'll have to point that this is necessary, but not sufficient for enabling an ever improving, extremely safe activity.

Aviation also have a just right amount of blame running in the system that is hard to replicate on any other area.

Post reply on HN