Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-Mortem for Google Compute Engine’s Global Outage on April 11

status.cloud.google.com

Post reply on HN
Print view View original

181–190 of 368 posts

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#181

Post by Gravityloss »

I'm waiting for the time when they push over the air updates to airplanes in flight.

"You can fly safely, we have canaries and staged deployment"

A year forward:

"Unfortunately because the canary verification as well as the staged deployment code was broken, instead of one crash and 300 dead, an update was pushed to all aircraft, which subsequently caused them to crash, killing 70,000 people."

I'm not 100% sure why they don't do the staged deployment for google scale server networking over a few days (or even weeks in some cases) instead of a few hours, but I don't know the details here...

It's good that they had manually triggerable configuration rollback possibility and a pre-set policy so it was solved so quickly.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#182

Post by jvolkman »

erichocean wrote: post #163
brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

Are BGP updates for Google's own router configurations really so frequent that they can't pay an engineer to at least monitor the propagation of configuration changes? In this case, a human would have instantly seen that the update was a) rejected (as explained in the postmortem), and b) holy shit, WHY DID THE ROUTER CHANGE ITS OWN CONFIGURATION TO BLOW AWAY ALL OF THE GCE ROUTES!?! I'm all for automation, but WTF? I…

The human part of any process will also eventually fail, and it's much more difficult to fix human bugs. Better to shoot for full automation.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#184

Post by tedks »

essayist wrote: post #148
gliderShip wrote: post #107

Earlier quoted context omitted.

Well No, There is an upper limit on the damage a bad driver can do by say crushing his car with a bus or something like that. Imagine a bug or malware triggered at the same moment world-wide. It could kill millions. So it not as simple as 'It just has to be better than a human'

I've been itching to release this terror movie plot into the wild: It's 2025 and more than 10% of the cars on the road in the US are self-driving. It's rush hour on a busy Friday afternoon in Washington, DC. Earlier that day, there'd been a handful of odd reports of self-driving Edsels (so as not to impugn an actual model) going haywire, and the NTSB has started its investigation. But then, at 430pm, highway patrol u…

This is a good movie plot, it just has one huge plot hole:

>It's rush hour on a busy Friday afternoon in Washington, DC.

>each phalanx moving towards the Clara Barton Parkway

>all converging

DC rush hour? Moving cars? Please. Independence Day made me suspend less disbelief.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#185

Post by ben_jones »

iopq wrote: post #146
lallysingh wrote: post #117

Earlier quoted context omitted.

It might lead to some decent work on formal verification of programs.

The problem is that perfectly-working software can easily get you in an accident. "Well, the bus didn't yield, and the programming said it would" It's not quite possible to write a car that avoids ALL accidents because a car has a speed and a turning radius and breaks only work so fast.

Still would be nice if the software could go "shit I'm about to get into an accident" and deploy air bags etc. that much faster. Plus now that I'm thinking about it, most accidents probably occur with some option that, had it been utilized, would've prevented the incident. Imagine the insane accident-avoiding swerve maneuvers a software program could potentially pull off.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#186

Post by takeda »

ambago wrote: post #92
reustle wrote: post #3

Earlier quoted context omitted.

I wouldn't worry so much. I'm sure self driving cars are going to save a lot more lives than they are going to end. Humans are terrible drivers, and the software will only get better.

Especially since "driver-error" is the cause of 94% of motor vehicle crashes in the U.S.[1], with 32,675 people killed and 2.3 million injured in 2014.[2] Worldwide, motor-vehicle crashes cause over 1.2 million deaths each year and are the leading cause of death for people between the ages of 15-29 years old.[3] It's estimated that self-driving cars could reduce vehicle crashes by approximately 90%! [4] [1] http://ww…

This is true for a single car, but self driving cars are introducing something that did not happen before. Imagine majority of cars are self driving cars and they all malfunctions due to bug in software update.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#187

Post by TeMPOraL »

iopq wrote: post #146
lallysingh wrote: post #117

Earlier quoted context omitted.

It might lead to some decent work on formal verification of programs.

The problem is that perfectly-working software can easily get you in an accident. "Well, the bus didn't yield, and the programming said it would" It's not quite possible to write a car that avoids ALL accidents because a car has a speed and a turning radius and breaks only work so fast.

> It's not quite possible to write a car that avoids ALL accidents because a car has a speed and a turning radius and breaks only work so fast.

It's not possible to have a self-driving car avoid all accidents, but one can presumably get pretty close. The realtime data from sensors give you enough information about the car itself, its surroundings and other objects around it to continuously compute a safety envelope - a subspace of the phase space of controllable parameters (like input, steering) within which the car can stop safely - and then make one of the goals to aggressively steer the car to remain in that envelope. This approach should be able to automagically handle things like safe driving distances or pedestrians suddenly running into the street.

Of course there will be a lot of details to account for when implementing this software, but it's important to realize that we have enough computing power to let the car continuously have every possible backup plan for almost any contingency in its electronic brain.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#188

Post by seanp2k2 »

xiphias wrote: post #162
sriram_sun wrote: post #141

Earlier quoted context omitted.

Car manufacturers conduct recalls all the time. There might be the possibility that a million self-driving cars will be held hostage from a remote control tower simultaneously leading to injury or death to millions. However, in practice, as soon as an issue is discovered, there will be the equivalent of recalls (remote updates) and things like this will be fixed. People who are uncomfortable with self driving cars wi…

People won't be able to override a buggy software: they can't even do that now, just look at the remote Audi and BMW hacks that can brake the car on the highway.

Or the jeep: https://blog.kaspersky.com/blackhat-jeep-cherokee-hack-expla...

Auto mfgs seem to be about 20-30 years behind when it comes to computers. Not really surprising that Tesla is whomping them on this front, given how SV people are scrambling to work there. You don't see that with the Big 3 or really any other car mfg.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#189

Post by blackflame7000 »

seanp2k2 wrote: post #175
rayval wrote: post #157

Earlier quoted context omitted.

Cool plot line, I'd go see that movie. A related scenario, one that theoretically could happen today, is hacking into commercial airliners auto-pilot systems, and directing dozens of flights onto a target. Set aside the fantasy movie plot angle, how realistic is this today? Is it any more or less plausible than the millions of cars scenario? If people are truly concerned about the car scenario, shouldn't they be worr…

In the airplane case, it's possible today: https://m.youtube.com/watch?v=CXv1j3GbgLk And https://m.youtube.com/watch?v=Uy3nXXZgqmg TL;DR you simulate a bunch of other planes in close proximity and the auto-pilot freaks out and tries to avoid them. As the second talk explains, the pilots would definitely notice and switch autopilot off. This is why IMO it's very important to not take ultimate control away from humans…

If a fuel injection system were to fail via fried component or even a short would trip a fuse and cause it to fail safe by cutting fuel and shutting off the car. Fuel Throttle cables however have definitely become stuck in their sheathing in the WOT position. Happened to my dad on the highway in a 1992 Rodeo Isuzu.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#190

Post by LeifCarrotson »

rayval wrote: post #157
essayist wrote: post #148

Earlier quoted context omitted.

I've been itching to release this terror movie plot into the wild: It's 2025 and more than 10% of the cars on the road in the US are self-driving. It's rush hour on a busy Friday afternoon in Washington, DC. Earlier that day, there'd been a handful of odd reports of self-driving Edsels (so as not to impugn an actual model) going haywire, and the NTSB has started its investigation. But then, at 430pm, highway patrol u…

Cool plot line, I'd go see that movie. A related scenario, one that theoretically could happen today, is hacking into commercial airliners auto-pilot systems, and directing dozens of flights onto a target. Set aside the fantasy movie plot angle, how realistic is this today? Is it any more or less plausible than the millions of cars scenario? If people are truly concerned about the car scenario, shouldn't they be worr…

I will disagree with the other commenter and say that this is more plausible for the aircraft than for the cars. Modern jetliners and military aircraft (scarrier yet) are purely fly-by-wire - there aren't cables running between the yokes and the control surfaces like in a Piper Cub, and if there were, no pilot would be strong enough to move them.

Yes, the autopilots can be turned off, but that's just a button, probably a button on the autopilot itself. Depending where the infection happens, the actual position of the yoke could be entirely ignored by the software. Or the motor controllers for the control surfaces themselves could be driving the plane, though I don't know how they could coordinate their actions and get feedback from an IMU.

Perhaps the pilots could rip out components and cut cables fast enough to prevent the plane from reaching its destination, and maybe they could tear out the affected component and limp back to a runway with what remains, but it's an entirely feasible movie plot.

But should we actually worry about either? No. The software sourcing, deployment and updating protocols at the various manufacturers of aircraft are certain to be secure. Right?

Post reply on HN