Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-Mortem for Google Compute Engine’s Global Outage on April 11

status.cloud.google.com

Post reply on HN
Print view View original

161–170 of 368 posts

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#161

Post by foota »

essayist wrote: post #148
gliderShip wrote: post #107

Earlier quoted context omitted.

Well No, There is an upper limit on the damage a bad driver can do by say crushing his car with a bus or something like that. Imagine a bug or malware triggered at the same moment world-wide. It could kill millions. So it not as simple as 'It just has to be better than a human'

I've been itching to release this terror movie plot into the wild: It's 2025 and more than 10% of the cars on the road in the US are self-driving. It's rush hour on a busy Friday afternoon in Washington, DC. Earlier that day, there'd been a handful of odd reports of self-driving Edsels (so as not to impugn an actual model) going haywire, and the NTSB has started its investigation. But then, at 430pm, highway patrol u…

Sounds vaugely similar to speed[1] (okay just in the sense that there's a vehicle that won't stop)

[1] https://en.wikipedia.org/wiki/Speed_(1994_film)

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#162

Post by xiphias »

sriram_sun wrote: post #141
Benjammer wrote: post #126

Earlier quoted context omitted.

If a bug can kill millions then it's not "better than a human" though, right?

Car manufacturers conduct recalls all the time. There might be the possibility that a million self-driving cars will be held hostage from a remote control tower simultaneously leading to injury or death to millions. However, in practice, as soon as an issue is discovered, there will be the equivalent of recalls (remote updates) and things like this will be fixed. People who are uncomfortable with self driving cars wi…

People won't be able to override a buggy software: they can't even do that now, just look at the remote Audi and BMW hacks that can brake the car on the highway.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#163

Post by erichocean »

brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

Are BGP updates for Google's own router configurations really so frequent that they can't pay an engineer to at least monitor the propagation of configuration changes? In this case, a human would have instantly seen that the update was a) rejected (as explained in the postmortem), and b) holy shit, WHY DID THE ROUTER CHANGE ITS OWN CONFIGURATION TO BLOW AWAY ALL OF THE GCE ROUTES!?!

I'm all for automation, but WTF? Insert even a semi-competent engineer in the loop to monitor the configuration change as it propagates around and the entire problem could have been addressed almost trivially, as the human engineers eventually decided to do.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#164

Post by bduerst »

gliderShip wrote: post #107
ben_jones wrote: post #64

Earlier quoted context omitted.

Self driving cars don't have to be perfect. They just have to be safer then driving is today [1]. The real question is if society can handle the unfairness that is death by random software error vs. death by negligent driving. It's easy to blame negligent driving on the driver, we're clearly not negligent so it really doesn't effect us right? But a software error might as well be an act of god, it's something that mi…

Well No, There is an upper limit on the damage a bad driver can do by say crushing his car with a bus or something like that. Imagine a bug or malware triggered at the same moment world-wide. It could kill millions. So it not as simple as 'It just has to be better than a human'

Sounds like current hardware defects and recalls for vehicles, with the difference being it's easier to fix.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#165

Post by noir_lord »

Afforess wrote: post #154
essayist wrote: post #148

Earlier quoted context omitted.

I've been itching to release this terror movie plot into the wild: It's 2025 and more than 10% of the cars on the road in the US are self-driving. It's rush hour on a busy Friday afternoon in Washington, DC. Earlier that day, there'd been a handful of odd reports of self-driving Edsels (so as not to impugn an actual model) going haywire, and the NTSB has started its investigation. But then, at 430pm, highway patrol u…

I'm afraid Daemon (novel) beat you to the punch. It's an excellent novel, about fairly similar situations. http://www.goodreads.com/book/show/6665847-daemon

One of my favourites, the sequels where similarly as good.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#166

Post by xiphias »

pveierland wrote: post #74
brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

It also showcases the great thing about self-driving cars. Even though accidents will happen, when it does there will be plenty of sensor data and logs which can be examined to find the exact cause in a post-mortem. An improvement to the software can then be made, and millions of cars deployed can all effectively learn from a single accident. With humans, the amount of knowledge gained and the collective improvement…

What does post-mortem mean in this context? The software one (after an accident) or the human one (after death)? I think It's crazy that the word gets back the original meaning..

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#167

Post by g8oz »

oldmanjay wrote: post #143
Grambo wrote: post #87

Earlier quoted context omitted.

Actually they have to be significantly safer than driving today. People would rather be unsafe and in control than not in control and a tiny bit safer. I know personally if a self driving car could only drive as well as I could then I'd still want to be the one driving.

People only have the illusion of safety when in control, and are also demonstrably incapable of judging their own ability to perform tasks. Your criterion won't be taken seriously by anyone involved in policy, because this is already well understood.

At a certain point the policy question will inevitably be: why should any regular person be even allowed to drive given the superior abilities of the machines? There are certain ideological assumptions that will then have to be debated. Making your own mistakes is a consequence of freedom. Limiting the freedom to make mistakes for the overall benefit to society is not uncontroversial - (see the gun control debate), and contributes to alienation in the Marxist sense of the term. There is more than utility involved here. Just because the trade-off doesn't matter to techno-determinists doesn't mean it it doesn't exist.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#168

Post by ocdtrekkie »

infinotize wrote: post #50
dreamcompiler wrote: post #19

Earlier quoted context omitted.

An auto-pilot for an airplane is a considerably easier problem to solve. No lanes; no pedestrians; very little other traffic; three spatial degrees of freedom. That's why auto-pilots for airplanes have existed for almost a century but we're just now beginning to get self-driving cars. Humans are still better at dealing with the full panoply of crap that road driving throws at us.

Aircraft autopilots also rely on experienced and licensed pilots to operate them and be responsible for the aircraft at all times. Self driving cars have assume the operator is not particularly capable nor paying attention to anything happening on the road.

Google Self-Driving Cars currently fail approximately every 1,500 miles, according to their own report. We are a LONG way away from being able to separate the operator's attention from driving.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#169

Post by kevan »

drstewart wrote: post #30
cjbprime wrote: post #17

It looks like there were at least three catastrophic bugs present: 1. Evaluated a configuration change before the change had finished syncing across all configuration files, resulting in rejecting the change. 2. So it tried to reject the change, but actually just deleted everything instead. 3. Something was supposed to catch changes that break everything, and it detected that everything was broken but its attempt to…

Seriously. This is a good postmortem, but these are hardly edge case bugs. In this case, major critical functionality just plain didn't work. Kind of shocking.

They explained the issues in laymans terms that most likely mask the true complexity of what happened. It's easy to read the final result: "tried to reject but then deleted everything" and think "Well duh that's bad, who would build a system that does that?", but I think you're fooling yourself if you think that edge cases couldn't cause that.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#170

Post by ocdtrekkie »

nxzero wrote: post #43
brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

Idea that edge cases in autonomous vehicles would result in 30,000+ deaths a year to me is a stretch. If you dispute this, please explain. If your position is that one death is too many, that is illogical relative to the option of letting people drive cars.

Currently, the self-driving software fails out on a Google Self-Driving Car every 1,500 miles. If the car suddenly stops trying to drive in the road, and the driver isn't attentive (or worse, if Google gets their way and convinces the laws to change so they don't have to have steering wheels) that's a lot of deaths.

I'm not saying it won't get better, but pretending self-driving cars is a cure-all right now is hilarious and insane.

Post reply on HN