Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-Mortem for Google Compute Engine’s Global Outage on April 11

status.cloud.google.com

Post reply on HN
Print view View original

171–180 of 368 posts

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#171

Post by ocdtrekkie »

jfoster wrote: post #20
brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

As long as the edge case bugs in self driving cars come up less frequently than human error, it's an overall improvement.

Currently, they don't. Google cars fail every 1,500 miles on average.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#172

Post by agildehaus »

kevindqc wrote: post #155
agildehaus wrote: post #149

Earlier quoted context omitted.

Google's cars drive millions of miles daily in simulation already. They take sensor data (including LIDAR) from past drives and essentially redrive their entire dataset with the updated software. Here's Google talking about it: https://static.googleusercontent.com/media/www.google.com/en... Of course this is only one component to feeling confident about pushing out an update where lives are on the line, but it's a ke…

But if a new update changes how the car drives, wouldn't the data that would've been captured by the updated car be different than what the old outdated car recorded? For example if the new update makes the car more aggressive, then other real drivers might be more careful, slow down more, etc compared to the original runs?

I'd assume the more severe the change the more they'd want to test in the real world.

And that once a comfortable self-driving experience is found, Google will not want to change it much.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#173

Post by pavel_lishin »

Afforess wrote: post #154
essayist wrote: post #148

Earlier quoted context omitted.

I've been itching to release this terror movie plot into the wild: It's 2025 and more than 10% of the cars on the road in the US are self-driving. It's rush hour on a busy Friday afternoon in Washington, DC. Earlier that day, there'd been a handful of odd reports of self-driving Edsels (so as not to impugn an actual model) going haywire, and the NTSB has started its investigation. But then, at 430pm, highway patrol u…

I'm afraid Daemon (novel) beat you to the punch. It's an excellent novel, about fairly similar situations. http://www.goodreads.com/book/show/6665847-daemon

And a really fun read. If someone here decides to get this book, get the sequel as well - Daemon ends on kind of a cliff-hanger.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#174

Post by cbr »

duskwuff wrote: post #25
pbreit wrote: post #16

Do SLAs even matter in the slightest? Or are they just sort of "feel-good" things or ways for negotiators to demonstrate their worth?

SLAs aren't about guaranteeing uptime. They're about setting consequences for downtime.

But once there are strong consequences for downtime the service provider is going to set up training, monitoring, oncall, etc to make sure things stay within the SLA limits. So you are effectively negotiating uptime.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#175

Post by seanp2k2 »

rayval wrote: post #157
essayist wrote: post #148

Earlier quoted context omitted.

I've been itching to release this terror movie plot into the wild: It's 2025 and more than 10% of the cars on the road in the US are self-driving. It's rush hour on a busy Friday afternoon in Washington, DC. Earlier that day, there'd been a handful of odd reports of self-driving Edsels (so as not to impugn an actual model) going haywire, and the NTSB has started its investigation. But then, at 430pm, highway patrol u…

Cool plot line, I'd go see that movie. A related scenario, one that theoretically could happen today, is hacking into commercial airliners auto-pilot systems, and directing dozens of flights onto a target. Set aside the fantasy movie plot angle, how realistic is this today? Is it any more or less plausible than the millions of cars scenario? If people are truly concerned about the car scenario, shouldn't they be worr…

In the airplane case, it's possible today: https://m.youtube.com/watch?v=CXv1j3GbgLk

And

https://m.youtube.com/watch?v=Uy3nXXZgqmg

TL;DR you simulate a bunch of other planes in close proximity and the auto-pilot freaks out and tries to avoid them. As the second talk explains, the pilots would definitely notice and switch autopilot off. This is why IMO it's very important to not take ultimate control away from humans in cars. I would personally never buy one of the Google (or any other) self-driving models with no controls. It already freaks me out that many cars are drive-by-wire (for the accelerator), and now even steer-by-wire: http://www.caranddriver.com/features/electric-feel-nissan-di... #noThankYouPlease

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#176

Post by pavel_lishin »

rayval wrote: post #157
essayist wrote: post #148

Earlier quoted context omitted.

I've been itching to release this terror movie plot into the wild: It's 2025 and more than 10% of the cars on the road in the US are self-driving. It's rush hour on a busy Friday afternoon in Washington, DC. Earlier that day, there'd been a handful of odd reports of self-driving Edsels (so as not to impugn an actual model) going haywire, and the NTSB has started its investigation. But then, at 430pm, highway patrol u…

Cool plot line, I'd go see that movie. A related scenario, one that theoretically could happen today, is hacking into commercial airliners auto-pilot systems, and directing dozens of flights onto a target. Set aside the fantasy movie plot angle, how realistic is this today? Is it any more or less plausible than the millions of cars scenario? If people are truly concerned about the car scenario, shouldn't they be worr…

> hacking into commercial airliners auto-pilot systems, and directing dozens of flights onto a target.

I would imagine that any pilot would figure out what was going on, unless it was on an incredibly foggy day.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#177

Post by seanp2k2 »

unprepare wrote: post #160
essayist wrote: post #148

Earlier quoted context omitted.

I've been itching to release this terror movie plot into the wild: It's 2025 and more than 10% of the cars on the road in the US are self-driving. It's rush hour on a busy Friday afternoon in Washington, DC. Earlier that day, there'd been a handful of odd reports of self-driving Edsels (so as not to impugn an actual model) going haywire, and the NTSB has started its investigation. But then, at 430pm, highway patrol u…

I can see it now - a modern day tiananmen square as a lone figure stands in front of a long line of Edsels

It's Will Smith, he's the president, and he just got finished dealing with a one of his super-cute children acting up, but actually doing something noble (but getting in trouble at school for it). Upon seeing the swarm: "Aww helllll naw"

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#178

Post by flurdy »

praxulus wrote: post #83
[deleted] wrote: post #57

Earlier quoted context omitted.

[deleted]

GCE downtime just means people lose money, it's not life-or-death. Skimping on QA in order to reduce costs and get to market faster is a perfectly reasonable decision when the consequences are so mundane.

I understand what you mean but that is generalising too much what people use GCE, public clouds, self hosted servers for, and especially going forward. It is not all convenience applications, game backends etc. What people these days use AWS/GCE for is so varied, even public sector use AWS Gov Region for example. Downtime consequences is not just money lost but can be life-and-death and for some application they need solid QA even if hosted in a public cloud.

It may (emphasise 'may') be how they share medical data via GCE/AWS that gets delayed just before a surgery (ok, edge case) or how they update bugs in a critical GPS model that happen to be used by an ambulance, or even a taxi used by pregnant lady that is about to drop, etc. Or simple general medical self diagnosis information site that by chance could have saved someone in that time slot. Or any other random non medical usage which involves a server and data of some kind that happen to be in GCE.

Yes critical real time systems often are on-premise or in self hosted data centres, but more and more are not especially if viewed as not critical but in some cases indirectly are critical.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#179

Post by seanp2k2 »

sriram_sun wrote: post #141
Benjammer wrote: post #126

Earlier quoted context omitted.

If a bug can kill millions then it's not "better than a human" though, right?

Car manufacturers conduct recalls all the time. There might be the possibility that a million self-driving cars will be held hostage from a remote control tower simultaneously leading to injury or death to millions. However, in practice, as soon as an issue is discovered, there will be the equivalent of recalls (remote updates) and things like this will be fixed. People who are uncomfortable with self driving cars wi…

Car manufacturers are some of the last people I trust to be doing software updates. The recent Takata airbag recall is an example of the ensuing fecal tornado from large recalls: http://blog.caranddriver.com/massive-takata-airbag-recall-ev...

In some cases, people are having to wait months to get new airbags because they just don't have them in stock. In the computer case, would you want to keep driving until they can get you scheduled for a software update? Remember that many cars can't update critical software OTA.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#180

Post by TeMPOraL »

unprepare wrote: post #160
essayist wrote: post #148

Earlier quoted context omitted.

I've been itching to release this terror movie plot into the wild: It's 2025 and more than 10% of the cars on the road in the US are self-driving. It's rush hour on a busy Friday afternoon in Washington, DC. Earlier that day, there'd been a handful of odd reports of self-driving Edsels (so as not to impugn an actual model) going haywire, and the NTSB has started its investigation. But then, at 430pm, highway patrol u…

I can see it now - a modern day tiananmen square as a lone figure stands in front of a long line of Edsels

Preventing the terror plot because the "do not drive over humans" goal overrides the "navigate to preselected target" goal?
Post reply on HN