Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Neural Net Tank Urban Legend

gwern.net

Post reply on HN
Print view View original

11–20 of 67 posts

Re: The Neural Net Tank Urban Legend

#11

Post by lgas »

Rotten194 wrote: post #5

I get the author's feelings about why we shouldn't tell this story, but I still disagree. It's a pithy, funny example of GIGO in machine learning. People could read conclusions about the abilities of neural networks from the story, but they're wrong to do so -- it's a PEBKAC error, not a technology one. "Truthy" cautionary tales are a near-universal feature of human cultures -- why shouldn't machine learning have som…

> I suggest that dataset bias is real but exaggerated by the tank story, giving a misleading indication of risks from deep learning and that it would be better to not repeat it but focus on established risks like AI systems optimizing for wrong utility functions.

He's not arguing against having cautionary tales, he's arguing that we should base them on actual problems instead of imaginary ones.

Re: The Neural Net Tank Urban Legend

#12

Post by semi-extrinsic »

gwern wrote: post #8
kainolophobia wrote: post #6

>I suggest that dataset bias is real but exaggerated by the tank story, giving a misleading indication of risks from deep learning I don't see how this story gives a "misleading" view of deep learning. From my (admittedly limited) experience with self-driving RC cars, this type of mistake is quite easy for a neural net to make while being quite difficult to detect. In our case, after utilizing a visual back-prop meth…

I assume you're referring to some simple lane-keeping CNN where the CNN predicts steering angle from a video recording+human inputs: and yes, your dataset isn't extensive enough, and you'll never have enough data either, not due to some amusing bias in your CNN or taking shortcuts, but because it's a reinforcement learning problem and not a classification problem - your RC CNN could learn a better model of the road w…

I did the human version of this when I was a newbie driver. I learned to predict traffic lights changing to red by watching the pedestrian signals as I approached an intersection. All the lights all over the city followed the same pattern. Then one day I happened upon one where the pattern was different, and I stopped for no reason at a green light, like an idiot.

Re: The Neural Net Tank Urban Legend

#13

Post by RodgerTheGreat »

I think the author's conclusion- that this scenario is unrealistic and would never happen given today's understanding of machine learning techniques- is extremely optimistic. NNs are demonstrably[1] not robust image classifiers.

In my opinion, it's far more dangerous to downplay the limitations of this technology and embolden snake-oil purveyors than it is to demand an inconvenient degree of rigor and caution in reporting results.

[1] https://arxiv.org/abs/1707.07397#

Re: The Neural Net Tank Urban Legend

#14

Post by haeffin »

gwern wrote: post #10
haeffin wrote: post #9

> a common preprocessing step in computer vision (and NNs in general) is to whiten the image by standardizing or transforming pixels to a normal distribution; this would tend to wipe global brightness levels, promoting invariance to illumination Is there anybody still doing this?

Seems to still be pretty common: https://scholar.google.com/scholar?hl=en&as_sdt=0%2C21&as_yl... Am I wrong?

"normalizing" is a bad search term, it can mean a lot of things. And whitening images is pretty much dead. What is done is subtracting mean colors from each pixel, but those are means over the whole database, not per-image, so that keeps brightness shifts intact.

Re: The Neural Net Tank Urban Legend

#15

Post by gwern »

RodgerTheGreat wrote: post #13

I think the author's conclusion- that this scenario is unrealistic and would never happen given today's understanding of machine learning techniques- is extremely optimistic . NNs are demonstrably[1] not robust image classifiers. In my opinion, it's far more dangerous to downplay the limitations of this technology and embolden snake-oil purveyors than it is to demand an inconvenient degree of rigor and caution in rep…

> I think the author's conclusion- that this scenario is unrealistic and would never happen given today's understanding of machine learning techniques- is extremely optimistic. NNs are demonstrably[1] not robust image classifiers.

I am well-aware of adversarial examples, and they are not the same thing as dataset bias, and I am very troubled by them. If you look at the section on whether we should tell the tank story as a cautionary story, I already say:

> I also fear that telling the tank story tends to promote complacency and underestimation of the state of the art by implying that NNs and AI in general are toy systems which are far from practicality and cannot work in the real world (particularly the story variants which date the tank story recently), or that such systems will fail in easily diagnosed and visible ways, ways which can be diagnosed by a human just comparing the photos or applying some political reasoning to the outputs, when what we actually see with deep learning are failure modes like "adversarial examples" which are quite as inscrutable as the neural nets themselves (or AlphaGo's one misjudged move resulting in its only loss to Lee Sedol).

To expand a little: dataset bias at least has the tendency to expose itself as soon as you try to apply it. You waste your time, but that's generally the worst part. I'm more worried about stuff like adversarial examples, which will work great in the field right up until a hacker comes by with a custom adversarial example (eg the adversarial car sign work showing you can trick simple CNNs into misclassifying speed limits and stop signs using adversarial examples pasted onto walls or signs or streets). This is not dataset bias; you can collect images of every single stop sign in the world and that will not stop adversarial examples.

> embolden snake-oil purveyors than it is to demand an inconvenient degree of rigor and caution in reporting results.

I think it's ironic to say that doing the very simplest level of fact-checking like 'did this story ever actually happen' is an 'inconvenient degree of rigor and caution' and 'emboldens snake-oil purveyors'.

Re: The Neural Net Tank Urban Legend

#16

Post by gwern »

haeffin wrote: post #14
gwern wrote: post #10

Earlier quoted context omitted.

Seems to still be pretty common: https://scholar.google.com/scholar?hl=en&as_sdt=0%2C21&as_yl... Am I wrong?

"normalizing" is a bad search term, it can mean a lot of things. And whitening images is pretty much dead. What is done is subtracting mean colors from each pixel, but those are means over the whole database, not per-image, so that keeps brightness shifts intact.

In searches you should err on the side of broadness. If you cut it down to just 'whitening', as you can see from the snippets as well, there are plenty of hits. You may not like whitening, but it does still seem to be common.

Re: The Neural Net Tank Urban Legend

#17

Post by 1024core »

All this speculation is silly. Just generate your own data set (since the story is from the early 90s, if not earlier, the number of training examples would have been quite small compared to today's data sets) and see if today's networks make the same mistake.

Re: The Neural Net Tank Urban Legend

#18

Post by Rotten194 »

lgas wrote: post #11
Rotten194 wrote: post #5

I get the author's feelings about why we shouldn't tell this story, but I still disagree. It's a pithy, funny example of GIGO in machine learning. People could read conclusions about the abilities of neural networks from the story, but they're wrong to do so -- it's a PEBKAC error, not a technology one. "Truthy" cautionary tales are a near-universal feature of human cultures -- why shouldn't machine learning have som…

> I suggest that dataset bias is real but exaggerated by the tank story, giving a misleading indication of risks from deep learning and that it would be better to not repeat it but focus on established risks like AI systems optimizing for wrong utility functions. He's not arguing against having cautionary tales, he's arguing that we should base them on actual problems instead of imaginary ones.

But ensuring correct datasets is a problem that people new to machine learning have to be made aware of. It's easy for people to see that NNs are good with 'noisy' data and incorrectly assume they can throw any data at it and get good results. And I think the quote is a false dilemma -- its not like we can't have multiple different stories for different problems. Make up / find a "truthy" story to spread for misoptimized utility functions if one doesn't exist, but there's no need to kill this useful story in the process.

Re: The Neural Net Tank Urban Legend

#20

Post by haeffin »

gwern wrote: post #16
haeffin wrote: post #14

Earlier quoted context omitted.

"normalizing" is a bad search term, it can mean a lot of things. And whitening images is pretty much dead. What is done is subtracting mean colors from each pixel, but those are means over the whole database, not per-image, so that keeps brightness shifts intact.

In searches you should err on the side of broadness. If you cut it down to just 'whitening', as you can see from the snippets as well, there are plenty of hits. You may not like whitening, but it does still seem to be common.

Using your search, at least for me, none of the snippets on the first page use normalization in the sense that you are in this context. So including that term just got you a lot of noise. And the only reference to whitening on that search page is not using it in an input pipeline, it is using ZCA to detect images that are modified to be adversarial.

If you want some better data than unreliable searches, go download pretrained models for popular architectures and popular frameworks and look at the input pipelines for them. You'll find that whitening is absolutely not common for image classification/detection today (yes, there are still some cases where it is used, but typically on smaller datasets where you can't get that invariance from data, which is the way you prefer it to be - if one class actually is more likely to be present in dark images, you don't want to kill that information).

Post reply on HN