Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

End to End Learning for Self-Driving Cars [pdf]

images.nvidia.com

Post reply on HN
Print view View original

51–60 of 102 posts

Re: End to End Learning for Self-Driving Cars [pdf]

#51

Post by joe_the_user »

rawnlq wrote: post #45
euyyn wrote: post #39

One problem with training a neural network end-to-end this way is that the system is susceptible to unpredictable glitches: The same principle that lets people trick a NN into [thinking a panda is a vulture]( https://codewords.recurse.com/issues/five/why-do-neural-netw... ) can happen randomly just by differing lighting/shadow conditions, sun glare, or who knows. One can always train the network with more and more sc…

For the "thinking a panda is a vulture" problem, don't humans fail in similar ways? The analogous examples for us are camouflage, optical illusions, logical fallacies, etc. It doesn't really have to be perfect as long as it doesn't fail in common scenarios.

For the "thinking a panda is a vulture" problem, don't humans fail in similar ways?

This is a good question. My impression is that humans fail and artificial neural networks fail but we don't know enough about the brain to say artificial neural networks fail in the same way as humans.

As another poster notes, humans accept human error more than computer error and I think that's because humans have an internal model of what other humans will do. If I see a car waving in a lane and going slowly, I have some ideas what's happening. I don't think that model would extend to a situation where neural network-driven car was acting "wonky".

Re: End to End Learning for Self-Driving Cars [pdf]

#52

Post by hooloovoo_zoo »

euyyn wrote: post #39

One problem with training a neural network end-to-end this way is that the system is susceptible to unpredictable glitches: The same principle that lets people trick a NN into [thinking a panda is a vulture]( https://codewords.recurse.com/issues/five/why-do-neural-netw... ) can happen randomly just by differing lighting/shadow conditions, sun glare, or who knows. One can always train the network with more and more sc…

I haven't seen a paper that shows this "tricking" can be used as a real world attack or happen randomly. Just because you can compute an input that has this unusual behavior doesn't mean there is a demonstrably nonzero probability of it happening.

Re: End to End Learning for Self-Driving Cars [pdf]

#53

Post by dkarapetyan »

hooloovoo_zoo wrote: post #52
euyyn wrote: post #39

One problem with training a neural network end-to-end this way is that the system is susceptible to unpredictable glitches: The same principle that lets people trick a NN into [thinking a panda is a vulture]( https://codewords.recurse.com/issues/five/why-do-neural-netw... ) can happen randomly just by differing lighting/shadow conditions, sun glare, or who knows. One can always train the network with more and more sc…

I haven't seen a paper that shows this "tricking" can be used as a real world attack or happen randomly. Just because you can compute an input that has this unusual behavior doesn't mean there is a demonstrably nonzero probability of it happening.

Wait? That's exactly what it means. Since the networks are not "continuous" you can't reason about how the system will behave in actual real world conditions because any random fluctuations can cause the whole thing to malfunction. I put continuous in quotes because it's not the real definition of continuous like in real analysis but a good enough analogy as in small variations in input should not lead to wildly different outputs.

This is why any model that lacks explanatory power can't be used in mission and safety critical systems. If it can't reason about things the same way people can reason about things then the system overall can't really be trusted. It's one thing when a translation from english to spanish is wrong, it's a completely another thing when the control software of a self-driving car decides to accelerate instead of break and the root cause analysis is people throwing their hands up and saying neural networks are inherently susceptible to these kinds of problems.

Re: End to End Learning for Self-Driving Cars [pdf]

#54

Post by argonaut »

dkarapetyan wrote: post #53
hooloovoo_zoo wrote: post #52

Earlier quoted context omitted.

I haven't seen a paper that shows this "tricking" can be used as a real world attack or happen randomly. Just because you can compute an input that has this unusual behavior doesn't mean there is a demonstrably nonzero probability of it happening.

Wait? That's exactly what it means. Since the networks are not "continuous" you can't reason about how the system will behave in actual real world conditions because any random fluctuations can cause the whole thing to malfunction. I put continuous in quotes because it's not the real definition of continuous like in real analysis but a good enough analogy as in small variations in input should not lead to wildly diff…

To be fair, you should be more precise. The attacks are specifically calculated. The combinatorial space of possible inputs is so massive that I'm sure it is extremely unlikely for a malicious input to occur randomly.

Re: End to End Learning for Self-Driving Cars [pdf]

#55

Post by aab0 »

euyyn wrote: post #39

One problem with training a neural network end-to-end this way is that the system is susceptible to unpredictable glitches: The same principle that lets people trick a NN into [thinking a panda is a vulture]( https://codewords.recurse.com/issues/five/why-do-neural-netw... ) can happen randomly just by differing lighting/shadow conditions, sun glare, or who knows. One can always train the network with more and more sc…

The adversarial examples are so weak that they disappear if you give the CNN even some attention or foveation mechanisms (that is, they work only on a single pass). How much effect are they going to have on a CNN being used at 30FPS+ to do lane following under constantly varying lighting and appearances and position? None.

Re: End to End Learning for Self-Driving Cars [pdf]

#56

Post by hooloovoo_zoo »

dkarapetyan wrote: post #53
hooloovoo_zoo wrote: post #52

Earlier quoted context omitted.

I haven't seen a paper that shows this "tricking" can be used as a real world attack or happen randomly. Just because you can compute an input that has this unusual behavior doesn't mean there is a demonstrably nonzero probability of it happening.

Wait? That's exactly what it means. Since the networks are not "continuous" you can't reason about how the system will behave in actual real world conditions because any random fluctuations can cause the whole thing to malfunction. I put continuous in quotes because it's not the real definition of continuous like in real analysis but a good enough analogy as in small variations in input should not lead to wildly diff…

These changes are not random. The whole reason neural networks work at all is because probable differences do not mislead the network.

Re: End to End Learning for Self-Driving Cars [pdf]

#57

Post by greeneggs »

EmployedRussian wrote: post #43
teraflop wrote: post #34

Earlier quoted context omitted.

Maybe you wouldn't consider this "credible", but Google publishes a monthly report listing every collision their autonomous cars have been involved in. [1] I was curious, so I went through the whole list. By my count, in the history of the program they've been involved in 19 accidents during autonomous operation, and the car was only at fault in one of those. [2] The majority of the other crashes were caused by other…

"While it was stopped" is the key. It's hard to argue that a car stopped at red light violates anyone's expectation of how a human would drive.

That's a misleading phrase. A number of the accidents have occurred because the Google car abruptly came to a stop in a situation where a human driver would not have stopped. Slamming on the brakes is dangerous.

Here is one example. It is hard to be sure exactly what happened, because Google obviously phrases its accident reports to put its cars in as favorable light as possible.

"April 28, 2016: A Google self-driving prototype vehicle travelling westbound in autonomous mode on Nita Avenue in Palo Alto was involved in an accident. The prototype vehicle came to a stop at the intersection of San Antonio Road, then, prior to making a right turn on San Antonio Road, began to gradually advance forward in order to get a better view of traffic approaching from the left on San Antonio Road. When the prototype vehicle stopped in order to yield to traffic approaching from the left on San Antonio Road, a vehicle approaching at approximately 9 mph from behind the prototype collided with the rear bumper of the prototype vehicle."

http://static.googleusercontent.com/media/www.google.com/en/...

Re: End to End Learning for Self-Driving Cars [pdf]

#58

Post by robbrown451 »

dpflan wrote: post #26
birdmanjeremy wrote: post #2

This is really interesting, but I'd want autonomous cars to be better than humans at driving, not to emulate them.

I think the trick is not to mix autonomous and non-autonomous. Cities delegate autonomous only zones where no non-autonomous cars can go; this creates a transportation circulatory system and safe experimental zone which can expand - which can include just making the area of city only for autonomous cars larger or gradual commingling with non-autonomous cars. Or again partial autonomy of cars on highways / more predic…

That seems like the trick to making sure they are never adopted in a big way.

I mean sure, that's an easy problem to solve but in that case why use cars at all and not people movers or the like?

Re: End to End Learning for Self-Driving Cars [pdf]

#59

Post by argonaut »

aab0 wrote: post #55
euyyn wrote: post #39

One problem with training a neural network end-to-end this way is that the system is susceptible to unpredictable glitches: The same principle that lets people trick a NN into [thinking a panda is a vulture]( https://codewords.recurse.com/issues/five/why-do-neural-netw... ) can happen randomly just by differing lighting/shadow conditions, sun glare, or who knows. One can always train the network with more and more sc…

The adversarial examples are so weak that they disappear if you give the CNN even some attention or foveation mechanisms (that is, they work only on a single pass). How much effect are they going to have on a CNN being used at 30FPS+ to do lane following under constantly varying lighting and appearances and position? None.

Are you referring to this foveation paper (http://arxiv.org/abs/1511.06292)? I'm quite skeptical of the claims in that paper; upon closer reading their experiments are problematic. Also, it appears the paper was rejected. I can elaborate if that is indeed the case.

Re: End to End Learning for Self-Driving Cars [pdf]

#60

Post by dkarapetyan »

hooloovoo_zoo wrote: post #56
dkarapetyan wrote: post #53

Earlier quoted context omitted.

Wait? That's exactly what it means. Since the networks are not "continuous" you can't reason about how the system will behave in actual real world conditions because any random fluctuations can cause the whole thing to malfunction. I put continuous in quotes because it's not the real definition of continuous like in real analysis but a good enough analogy as in small variations in input should not lead to wildly diff…

These changes are not random. The whole reason neural networks work at all is because probable differences do not mislead the network.

This is not true at all. Adversarial input can indeed be probable input depending on your definitions and I haven't seen anything yet that describes the probability distributions of inputs. Everyone takes a bunch of training examples and extrapolates from there.
Post reply on HN