Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-Mortem for Google Compute Engine’s Global Outage on April 11

status.cloud.google.com

Post reply on HN
Print view View original

191–200 of 368 posts

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#191

Post by blackflame7000 »

pavel_lishin wrote: post #176
rayval wrote: post #157

Earlier quoted context omitted.

Cool plot line, I'd go see that movie. A related scenario, one that theoretically could happen today, is hacking into commercial airliners auto-pilot systems, and directing dozens of flights onto a target. Set aside the fantasy movie plot angle, how realistic is this today? Is it any more or less plausible than the millions of cars scenario? If people are truly concerned about the car scenario, shouldn't they be worr…

> hacking into commercial airliners auto-pilot systems, and directing dozens of flights onto a target. I would imagine that any pilot would figure out what was going on, unless it was on an incredibly foggy day.

It's Hollywood, name one movie where the villain did not disable the manual override. That's villainy 101.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#192

Post by nxzero »

Gravityloss wrote: post #181

I'm waiting for the time when they push over the air updates to airplanes in flight. "You can fly safely, we have canaries and staged deployment" A year forward: "Unfortunately because the canary verification as well as the staged deployment code was broken, instead of one crash and 300 dead, an update was pushed to all aircraft, which subsequently caused them to crash, killing 70,000 people." I'm not 100% sure why t…

Comparing the risk of a live update to a system lives depend on to the risk of some Google services going down is irrational.

At some point, delaying the deployment of updates system wide would cause more, not less risks.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#193

Post by TeMPOraL »

oldmanjay wrote: post #143
Grambo wrote: post #87

Earlier quoted context omitted.

Actually they have to be significantly safer than driving today. People would rather be unsafe and in control than not in control and a tiny bit safer. I know personally if a self driving car could only drive as well as I could then I'd still want to be the one driving.

People only have the illusion of safety when in control, and are also demonstrably incapable of judging their own ability to perform tasks. Your criterion won't be taken seriously by anyone involved in policy, because this is already well understood.

While I agree with your first sentence, I think you're ignoring the fact that when media get wind of a case like this, it's almost only the irrational opinions of masses that matter. In western democracies, politics - and thus policies - is driven by pandering to the population (and bribes^Wlobbying).

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#194

Post by mitchellhislop »

pavel_lishin wrote: post #173
Afforess wrote: post #154

Earlier quoted context omitted.

I'm afraid Daemon (novel) beat you to the punch. It's an excellent novel, about fairly similar situations. http://www.goodreads.com/book/show/6665847-daemon

And a really fun read. If someone here decides to get this book, get the sequel as well - Daemon ends on kind of a cliff-hanger.

Freedom (TM) is the sequel, and the author (Daniel Suarez) has a few other near-term what-if-this-all-goes-skynet books which are equally good.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#195

Post by kentonv »

Gravityloss wrote: post #181

I'm waiting for the time when they push over the air updates to airplanes in flight. "You can fly safely, we have canaries and staged deployment" A year forward: "Unfortunately because the canary verification as well as the staged deployment code was broken, instead of one crash and 300 dead, an update was pushed to all aircraft, which subsequently caused them to crash, killing 70,000 people." I'm not 100% sure why t…

The answer, of course, is that slower and less-frequent deployments mean slower progress building a better platform and delivering new features. If breakages could lead to plane crashes then, obviously, we'd want them to slow down. But if it mainly means no one can listen to Spotify for 15 minutes then that calls for a different trade-off.

As a founder of a startup that hosts services on GCE I'm happy with the trade-off they've chosen.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#196

Post by flippyhead »

ben_jones wrote: post #64
brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

Self driving cars don't have to be perfect. They just have to be safer then driving is today [1]. The real question is if society can handle the unfairness that is death by random software error vs. death by negligent driving. It's easy to blame negligent driving on the driver, we're clearly not negligent so it really doesn't effect us right? But a software error might as well be an act of god, it's something that mi…

Yeah and it's funny how people seem to have some level of tolerance for death-by-design-flaws-in–hardware but somehow, software is a different kind of engineering endeavor. My guess is this will persist for some time, but eventually even out.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#197

Post by seanp2k2 »

iopq wrote: post #146
lallysingh wrote: post #117

Earlier quoted context omitted.

It might lead to some decent work on formal verification of programs.

The problem is that perfectly-working software can easily get you in an accident. "Well, the bus didn't yield, and the programming said it would" It's not quite possible to write a car that avoids ALL accidents because a car has a speed and a turning radius and breaks only work so fast.

To other replies, since we've reached max depth: yes, and that's why it makes no sense to me to equip such badly-spec'd vehicles with the self-driving bits first (Tesla S excluded). I drive a higher-than-normal performance vehicle for the exact reason that it gives me more options when I need to get out of a bad situation. I can out-break, out-swerve (more lateral grip), and out-accelerate most other cars on the road. The stopping distance (factor of tire grip and break power) has definitely helped me avoid accidents. Lower body roll = more grip during more extreme maneuvers + more control during them.

Top Gear discussed speed limits being set based on worst-case breaking distances. Here's an analysis of that: http://www.jmp.co.uk/forward-thinking/update/top-gear-and-sp... but TL;DR performance vehicles can be safer since they're more capable.

Think what an F1 or rally car could do with computers driving it and avoiding accidents.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#198

Post by seanp2k2 »

pjlegato wrote: post #123
deegles wrote: post #106

Earlier quoted context omitted.

This would have a dramatic chilling effect on hiring for self-driving car software developers, which would ironically make them less safe.

For this reason, most self-driving car development will only happen at large companies like Google, Tesla, Ford, etc. because they are the only ones who will be able to afford to purchase a massive general liability insurance policy.

I think that the other part with insurance is that insurers have no idea what the risk involved actually is (since it's not been around long) so they aim way high to cover themselves.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#199

Post by mikx007 »

brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

Assuming bugs are never intentional and mostly random... Maybe instead of one autopilot software, self driving cars of the future will have several, developed by completely different teams. Then a self driving car can take some sort of average or most common output instruction (thus minimizing the risk of random bugs/edge cases...etc.)

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#200

Post by jlgaddis »

Sanddancer wrote: post #11
wyldfire wrote: post #5

> . Internal monitors generated dozens of alerts in the seconds after the traffic loss became visible at 19:08 ... revert the most recent configuration changes ... the time from detection to decision to revert to the end of the outage was thus just 18 minutes. It's certainly good that they detected it as fast as they did. But I wonder if the fix time could be improved upon? Was the majority of that time spent discuss…

From the rest of the post, it sounds like replication time. Datacenters started dropping an hour beforehand one by one, and they had all fallen over by 19:08. Given that you have to push the rollback to routers around the world, and that peer routers have to propagate the changes from there, 18 minutes for a change like this sounds about right.

... although once the first datacenter once again announced the prefixes into BGP, those networks would have been reachable again, from everywhere. I imagine this is what happened at 19:27 -- the first datacenter came back online.

Of course, the traffic load might have overwhelmed that single datacenter but that would be alleviated as soon as additional datacenters came back online ("announced the prefixes"). A portion of the traffic load would shift to each new datacenter as it came back online.

It could have been hours later before they were all operational again but, as far as the users were concerned, the service was up and running and back to normal as soon as the first one or two datacenters came back up.

Post reply on HN