Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-Mortem for Google Compute Engine’s Global Outage on April 11

status.cloud.google.com

Post reply on HN
Print view View original

241–250 of 368 posts

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#241

Post by [deleted] »

amazon_not wrote: post #150
amelius wrote: post #104

Earlier quoted context omitted.

> Part of why human drivers are allowed today is because the people who bear the cost of driving decisions are directly involved in making those decisions. This gives me weird visions of Google engineers with a necklace that explodes in the event that one of their cars causes an accident :S

I wonder if you've read Fallen Dragon by Peter F. Hamilton? Unremovable, remote controllable lethal necklaces are a central plot device in the mercenary invasion. They are put on randomly chosen civilians as "collateral" to ensure co-operation and disincentive insurgency.

[deleted]

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#242

Post by unprepare »

TeMPOraL wrote: post #180
unprepare wrote: post #160

Earlier quoted context omitted.

I can see it now - a modern day tiananmen square as a lone figure stands in front of a long line of Edsels

Preventing the terror plot because the "do not drive over humans" goal overrides the "navigate to preselected target" goal?

my first thought actually was, how would the cars react to road spikes thrown by police? Especially on a highway with barriers on either side.

Of course a competent writer would've thrown in a line about how these cars are on run flats at some point...

Our only hope is for the scientists in the So-Secret-President-Doesnt-Even-Know Facility to come up with something so crazy it just might work

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#243

Post by rrdharan »

halayli wrote: post #218

This isn't the first time a config system at Google causes a major outage. https://googleblog.blogspot.com/2014/01/todays-outage-for-se...

That's entirely unsurprising. The recent major Facebook outage was also caused by bad configuration IIRC.

See: http://danluu.com/postmortem-lessons/

> Configuration > > Configuration bugs, not code bugs, are the most common cause > I’ve seen of really bad outages. When I looked at publicly available > postmortems, searching for “global outage postmortem” returned > about 50% outages caused by configuration changes. Publicly > available postmortems aren’t a representative sample of all > outages, but a random sampling of postmortem databases also > reveals that config changes are responsible for a disproportionate > fraction of extremely bad outages. As with error handling, I’m > often told that it’s obvious that config changes are scary, but > it’s not so obvious that most companies test and stage config > changes like they do code changes.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#244

Post by profeta »

brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

why do you think that will wait for cars? https://en.wikipedia.org/wiki/Therac-25

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#245

Post by wefarrell »

llamataboot wrote: post #69
ben_jones wrote: post #64

Earlier quoted context omitted.

Self driving cars don't have to be perfect. They just have to be safer then driving is today [1]. The real question is if society can handle the unfairness that is death by random software error vs. death by negligent driving. It's easy to blame negligent driving on the driver, we're clearly not negligent so it really doesn't effect us right? But a software error might as well be an act of god, it's something that mi…

No because negligent driving doesn't just put the driver at risk - it puts everyone else on the road plus pedestrians at risk

Exactly, negligent drivers don't just kill themselves and there is very little you can do to prevent one from killing you.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#246

Post by 72deluxe »

creshal wrote: post #213
seanp2k2 wrote: post #179

Earlier quoted context omitted.

Car manufacturers are some of the last people I trust to be doing software updates. The recent Takata airbag recall is an example of the ensuing fecal tornado from large recalls: http://blog.caranddriver.com/massive-takata-airbag-recall-ev... In some cases, people are having to wait months to get new airbags because they just don't have them in stock. In the computer case, would you want to keep driving until they ca…

A friend of mine works for VW's engine computer division. Yes, those engine computers. After all I've heard of their development methods (or lack thereof), I'm surprised the engines even start more often than one time out of ten.

My VW Golf has a bug where the driver's side door will be completely unresponsive after starting the ignition, with all the lights on the door being off too. After 5 to 10 seconds it will become responsive, which is a bit annoying if you're trying to open the windows to clear the damp mist on them, as you can't....

Also, if during normal routine you run through all four electric windows to close them (so passenger, driver, passenger rear, driver rear) in that order, you hear the solenoids click in a COMPLETELY different order. I am not sure if it is prioritising the messages in some way but the order that the windows "click" is not the order I press the buttons.

Also, I can get the CD player to crash.

Such minor noticeable issues make me think about the quality of the more important bits somewhat.

The breakdown on Toyota's safety code was interesting; and frightening really.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#248

Post by landryraccoon »

seanp2k2 wrote: post #175
rayval wrote: post #157

Earlier quoted context omitted.

Cool plot line, I'd go see that movie. A related scenario, one that theoretically could happen today, is hacking into commercial airliners auto-pilot systems, and directing dozens of flights onto a target. Set aside the fantasy movie plot angle, how realistic is this today? Is it any more or less plausible than the millions of cars scenario? If people are truly concerned about the car scenario, shouldn't they be worr…

In the airplane case, it's possible today: https://m.youtube.com/watch?v=CXv1j3GbgLk And https://m.youtube.com/watch?v=Uy3nXXZgqmg TL;DR you simulate a bunch of other planes in close proximity and the auto-pilot freaks out and tries to avoid them. As the second talk explains, the pilots would definitely notice and switch autopilot off. This is why IMO it's very important to not take ultimate control away from humans…

> I would personally never buy one of the Google (or any other) self-driving models with no controls.

It won't matter if all the other cars on the road besides yours don't have controls.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#249

Post by pacaro »

unprepare wrote: post #242
TeMPOraL wrote: post #180

Earlier quoted context omitted.

Preventing the terror plot because the "do not drive over humans" goal overrides the "navigate to preselected target" goal?

my first thought actually was, how would the cars react to road spikes thrown by police? Especially on a highway with barriers on either side. Of course a competent writer would've thrown in a line about how these cars are on run flats at some point... Our only hope is for the scientists in the So-Secret-President-Doesnt-Even-Know Facility to come up with something so crazy it just might work

An extremely selective and directional EMP cannon springs to mind as being sufficiently Hollywood

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#250

Post by lallysingh »

elcapitan wrote: post #153
lallysingh wrote: post #117

Earlier quoted context omitted.

It might lead to some decent work on formal verification of programs.

Is there actually some way of formal verification of software that is driven mostly by machine learning? I imagine there's a small core of code that runs some trained models, but how are they being formally verified? How do we know there's not a blind spot in that model that turns out to be fatal under certain conditions?

I'd imagine that you'd have some hybrid of ML and traditional code, and may be able to reason statistically about the ML sections, and user traditional (verified) code to cut the tail on the distribution.

All pipe dreams of mine, but the research potential here could be worth flaming truckloads of grant money :-)

Post reply on HN