Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-Mortem for Google Compute Engine’s Global Outage on April 11

status.cloud.google.com

Post reply on HN
Print view View original

101–110 of 368 posts

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#101

Post by raverbashing »

VonGuard wrote: post #6
reustle wrote: post #3

Earlier quoted context omitted.

I wouldn't worry so much. I'm sure self driving cars are going to save a lot more lives than they are going to end. Humans are terrible drivers, and the software will only get better.

Yeah, remember, auto-pilot in a plane needs to be 100% reliable, or everyone dies. A car needs to be, I dunno, 80%? Compared to a bad human driver, who still drives every damn day, a computer need only be about 60% reliable to be better. People suck at driving. Even a shitty self-driving car will save a ton of lives simply by obeying traffic laws.

> auto-pilot in a plane needs to be 100% reliable, or everyone dies

Actually it is much more simpler than a self-driving car. And if there is a problem it disengages.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#102

Post by wyldfire »

clebio wrote: post #40
wyldfire wrote: post #5

> . Internal monitors generated dozens of alerts in the seconds after the traffic loss became visible at 19:08 ... revert the most recent configuration changes ... the time from detection to decision to revert to the end of the outage was thus just 18 minutes. It's certainly good that they detected it as fast as they did. But I wonder if the fix time could be improved upon? Was the majority of that time spent discuss…

This sounds like hindsight bias.

Well, perhaps, but -- to be clear, I'm not suggesting that there is a problem, rather gathering more information in order to determine whether there's a more optimal solution.

e.g. if the detection mechanism latency is ~60s but the time-to-resolve is 18 mins, then I wonder: "how good could the best possible recovery system be?" Implicit in this question is that I think the answer to my question could just as easily be "19 minutes" as it could "5 minutes."

It's not a bias if I'm asking questions in order to improve the system. Could this fault have been predicted? Yes, IMO it could have. I believe that the fault in this case is grossly summarized as "rollback fails to rollback."

What if the major driver of the 18 minute latency was getting the right humans to agree that "execute recovery plan Q" was the right move? If that were the case then perhaps another item to learn could be "recovery policy item 23: when 'rollback fails to rollback', summon at least 3 of 5 Team Z participants and get consensus on recovery plan." And then maybe there could be a corresponding "change policy item 54: changes shall be barred until/unless 5 participants of Team Z are 'available'"

But that's all moot, if "fastest possible recovery [given XYZ constraints of BGP or whatever system] is ~16 minutes." Which it sounds like may indeed be the case.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#103

Post by heisenbit »

"Lessons learned from reading post-mortems" http://danluu.com/postmortem-lessons/ is a good place to dig deeper

The first graph quoted from a survey paper is a classic fitting the GCE outage well:

Initial error --92%--> Incorrect handling of errors explicitly signaled in software

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#104

Post by amelius »

ThrustVectoring wrote: post #88
794CD01 wrote: post #80

Earlier quoted context omitted.

They don't have to be safer than driving is today. They can be significantly less safe while still being an improvement for society because drivers will be able to focus on other activities while travelling instead of wasting that time focusing on driving the car.

There's psychological and game theoretic factors that the safety has to overcome in order to be acceptable. Part of why human drivers are allowed today is because the people who bear the cost of driving decisions are directly involved in making those decisions. Once you give up control to a third party, they need to be significantly better to make it an acceptable choice on the individual level. In other words, I agr…

> Part of why human drivers are allowed today is because the people who bear the cost of driving decisions are directly involved in making those decisions.

This gives me weird visions of Google engineers with a necklace that explodes in the event that one of their cars causes an accident :S

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#106

Post by deegles »

xanderjanz wrote: post #96
ben_jones wrote: post #64

Earlier quoted context omitted.

Self driving cars don't have to be perfect. They just have to be safer then driving is today [1]. The real question is if society can handle the unfairness that is death by random software error vs. death by negligent driving. It's easy to blame negligent driving on the driver, we're clearly not negligent so it really doesn't effect us right? But a software error might as well be an act of god, it's something that mi…

I imagine we'd treat it like we do getting hit by a drunk driver. Vilify reckless programmers, and don't think about it.

This would have a dramatic chilling effect on hiring for self-driving car software developers, which would ironically make them less safe.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#107

Post by gliderShip »

ben_jones wrote: post #64
brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

Self driving cars don't have to be perfect. They just have to be safer then driving is today [1]. The real question is if society can handle the unfairness that is death by random software error vs. death by negligent driving. It's easy to blame negligent driving on the driver, we're clearly not negligent so it really doesn't effect us right? But a software error might as well be an act of god, it's something that mi…

Well No, There is an upper limit on the damage a bad driver can do by say crushing his car with a bus or something like that. Imagine a bug or malware triggered at the same moment world-wide. It could kill millions. So it not as simple as 'It just has to be better than a human'

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#108

Post by magicalist »

manquer wrote: post #72
mjibson wrote: post #27

Earlier quoted context omitted.

I'm attempting to even imagine how one would build a useful way to test this. Would they have to have a secondary, world-wide datacenter network with all their various services behind it?

While testing would have been quite difficult, any simple canary release or timed release mechanism would have prevented this / limited the damage. At such mission critical systems, applying any global change in a such manner is asking for it, Devops can also be SPOF, this seems one such case.

There was a canary release.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#109

Post by tacon »

brianwawok wrote: post #33
reustle wrote: post #3

Earlier quoted context omitted.

I wouldn't worry so much. I'm sure self driving cars are going to save a lot more lives than they are going to end. Humans are terrible drivers, and the software will only get better.

I know this logically. But emotionally I know how many bugs I have written in my life. I know software devs are human.. aka I know how the sausage is made.

I had the same feeling just before Y2K. Too much knowledge of the process, it all must work the first time in production, etc., etc. I was pleasantly surprised at the non-event of Y2K.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#110

Post by Dylan16807 »

brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

> As I assumed it was kind of a corner case bug meet corner case bug met corner case bug.

I wouldn't really agree with that. There were two pieces of code designed to perform checks on new configs and cancel them. They both failed. Neither of those checks is a corner case. If you had a spec sheet for the system that manages IP blocks, that functionality would be listed as a feature right up front.

Post reply on HN