Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-Mortem for Google Compute Engine’s Global Outage on April 11

status.cloud.google.com

Post reply on HN
Print view View original

91–100 of 368 posts

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#91

Post by brianwawok »

kohanz wrote: post #49
brianwawok wrote: post #33

Earlier quoted context omitted.

I know this logically. But emotionally I know how many bugs I have written in my life. I know software devs are human.. aka I know how the sausage is made.

Yes, but have you been part of the development & testing effort for mission-critical software (e.g. a class 1 or 2 medical device?). It's not true in all cases, but for the most part the level of QA that goes into the devices before release is significantly higher than that of your average product. This is why regulation is required.

No, but I wrote software for financial exchanges that made the newspaper when I wrote bugs. And I did write bugs ;)

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#92

Post by ambago »

reustle wrote: post #3
brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

I wouldn't worry so much. I'm sure self driving cars are going to save a lot more lives than they are going to end. Humans are terrible drivers, and the software will only get better.

Especially since "driver-error" is the cause of 94% of motor vehicle crashes in the U.S.[1], with 32,675 people killed and 2.3 million injured in 2014.[2] Worldwide, motor-vehicle crashes cause over 1.2 million deaths each year and are the leading cause of death for people between the ages of 15-29 years old.[3]

It's estimated that self-driving cars could reduce vehicle crashes by approximately 90%! [4]

[1] http://www-nrd.nhtsa.dot.gov/pubs/812115.pdf [2] http://www-nrd.nhtsa.dot.gov/Pubs/812219.pdf [3] http://www.who.int/violence_injury_prevention/road_safety_st... [4] http://www.mckinsey.com/industries/automotive-and-assembly/o...

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#93

Post by brianwawok »

bcook wrote: post #52
brianwawok wrote: post #38

Earlier quoted context omitted.

So crashes like that include drunk drivers, drugged drivers, and texting drivers. What is the crashes per miles for a paying attention driver? If it is 1 per million miles, the self driving car would need to be a lot lower. Now if it was 4am and I am falling asleep at the wheel, I bet any self driving car would beat me. So cool to turn on, but maybe not for a daytime cruise...

Why are you comparing self-driving cars to exclusively a "paying attention driver"? For self-driving cars to be safer than human drivers, there is no requirement that the self-driving cars should be better/safer than the best human driver... the self-driving car simply needs to be safer than the majority of humans.

> For self-driving cars to be safer than human drivers, there is no requirement that the self-driving cars should be better/safer than the best human driver... the self-driving car simply needs to be safer than the majority of humans.

That is true on a whole, but not true for ME. It needs to be safer than ME, not some hypotehtical average person.

Further compounding it:

> For driving skills, 93% of the U.S. sample and 69% of the Swedish sample put themselves in the top 50% [0]

0 - https://en.wikipedia.org/wiki/Illusory_superiority

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#94

Post by yeukhon »

brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

Airplanes are equipped with software and many pilots would turn on auto-pilots after a long take off. Bugs are everywhere, and it's just a matter of time before one is so critical and kill people. So our best bet is better quality assurance through proof and overtesting (do this incrementally!)

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#95

Post by [deleted] »

ben_jones wrote: post #64
brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

Self driving cars don't have to be perfect. They just have to be safer then driving is today [1]. The real question is if society can handle the unfairness that is death by random software error vs. death by negligent driving. It's easy to blame negligent driving on the driver, we're clearly not negligent so it really doesn't effect us right? But a software error might as well be an act of god, it's something that mi…

[deleted]

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#96

Post by xanderjanz »

ben_jones wrote: post #64
brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

Self driving cars don't have to be perfect. They just have to be safer then driving is today [1]. The real question is if society can handle the unfairness that is death by random software error vs. death by negligent driving. It's easy to blame negligent driving on the driver, we're clearly not negligent so it really doesn't effect us right? But a software error might as well be an act of god, it's something that mi…

I imagine we'd treat it like we do getting hit by a drunk driver. Vilify reckless programmers, and don't think about it.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#97

Post by balls187 »

daveguy wrote: post #42
balls187 wrote: post #10

Nice post mortem. That outtage gives GCE at best a four 9's reliability for 2016.

Based on the higher level status page: https://status.cloud.google.com/summary It looks like GCE uptime is well below four 9's reliability for a sliding 1 year timeframe.

April's incident is unique, This was the only case (listed) that was a service outtage, which impacted all of GCE.

The other incidents (as far as I can tell), were service disruptions at the AZ/regional level. Those disruptions don't impact the 9's, as GCE was available for other regions.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#98

Post by stcredzero »

In this event, the canary step correctly identified that the new configuration was unsafe. Crucially however, a second software bug in the management software did not propagate the canary step’s conclusion back to the push process, and thus the push system concluded that the new configuration was valid and began its progressive rollout.

Classic Two Generals. "No news is good news," generally isn't a good design philosophy for systems designed to detect trouble. How do we know that stealthy ninjas haven't assassinated our sentries? Well, we haven't heard anything wrong...

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#99

Post by [deleted] »

teraflop wrote: post #31

> There are a number of lessons to be learned from this event -- for example, that the safeguard of a progressive rollout can be undone by a system designed to mask partial failures -- ... This is a really important point that should be more generally known. To quote Google's own "Paxos Made Live" paper, from 2007: > In closing we point out a challenge that we faced in testing our system for which we have no systemat…

[deleted]

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#100

Post by strictfp »

teraflop wrote: post #31

> There are a number of lessons to be learned from this event -- for example, that the safeguard of a progressive rollout can be undone by a system designed to mask partial failures -- ... This is a really important point that should be more generally known. To quote Google's own "Paxos Made Live" paper, from 2007: > In closing we point out a challenge that we faced in testing our system for which we have no systemat…

Degraded modes of operation is one example of how to visualize masked errors. Another is to trigger an alarm on fallbacks.

As a general reflection, many distributed system leave out the cause of their changes and only log actions. Instead of logging "new membership, new members are b,c,d" you are better of logging "node a has not responded to heartbeat in the last 30 seconds, considering it faulty". Following such a principle makes it much easier to spot masked bugs, since you can reason about the behaviour much better.

Aggregating logs to a central location and being able to analyze global behaviour in retrospect is also a great feature.

Post reply on HN