Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Neural Net Tank Urban Legend

gwern.net

Post reply on HN
Print view View original

21–30 of 67 posts

Re: The Neural Net Tank Urban Legend

#21

Post by goialoq »

Rotten194 wrote: post #18
lgas wrote: post #11

Earlier quoted context omitted.

> I suggest that dataset bias is real but exaggerated by the tank story, giving a misleading indication of risks from deep learning and that it would be better to not repeat it but focus on established risks like AI systems optimizing for wrong utility functions. He's not arguing against having cautionary tales, he's arguing that we should base them on actual problems instead of imaginary ones.

But ensuring correct datasets is a problem that people new to machine learning have to be made aware of. It's easy for people to see that NNs are good with 'noisy' data and incorrectly assume they can throw any data at it and get good results. And I think the quote is a false dilemma -- its not like we can't have multiple different stories for different problems. Make up / find a "truthy" story to spread for misoptim…

If a failure mode has never been reported in the wild, why is it so important to tell a juicy story about it, at the expense of attention to empirically observed failure modes?

Re: The Neural Net Tank Urban Legend

#22

Post by gok »

I always heard the version that went the other way around. After it was shown that single layer perceptrons were unable to deal with data sets that weren't linearly separable, there was an effort to figure out how the single layer tank classifier was working.

Re: The Neural Net Tank Urban Legend

#23

Post by robotresearcher »

Rotten194 wrote: post #5

I get the author's feelings about why we shouldn't tell this story, but I still disagree. It's a pithy, funny example of GIGO in machine learning. People could read conclusions about the abilities of neural networks from the story, but they're wrong to do so -- it's a PEBKAC error, not a technology one. "Truthy" cautionary tales are a near-universal feature of human cultures -- why shouldn't machine learning have som…

It's a plausible story. Fine to present as a parable, but we should stop presenting it as true unless we can find a reliable source for it.

Until today, I believed it was true. It was told to me as an undergrad, by a professor who believed it himself.

Re: The Neural Net Tank Urban Legend

#25

Post by KVFinn »

tomelders wrote: post #24
andreasvc wrote: post #19

For a better, actual example of this problem, see the leopard sofa: http://rocknrollnerd.github.io/ml/2015/05/27/leopard-sofa.ht...

That comment section took an immediate and unexpected turn for the worse.

>That comment section took an immediate and unexpected turn for the worse.

What the heck is going on there?

Re: The Neural Net Tank Urban Legend

#26

Post by eric_h »

KVFinn wrote: post #25
tomelders wrote: post #24

Earlier quoted context omitted.

That comment section took an immediate and unexpected turn for the worse.

>That comment section took an immediate and unexpected turn for the worse. What the heck is going on there?

Terry Davis - he occasionally chimes in here with similarly themed posts (but only if you have show dead enabled).

He's schizophrenic, is famous for TempleOS and infamous for the contents of his posts on the internet.

Re: The Neural Net Tank Urban Legend

#27

Post by akavel »

Umm, but then a story linked from the article as "alternative example" (thus presumably "better" than the tank story), and it being one from HN by the way, seems to have a nearly identical gist, at least for me as a layman: https://news.ycombinator.com/item?id=6269114 - only not about neural nets, but genetic/evolutionary algorithms. Or is it somehow drastically different and I just don't understand that?

Re: The Neural Net Tank Urban Legend

#28

Post by teraflop »

I first encountered this idea in a sci-fi story (I want to say it was one of Peter Watts' "Rifters" novels, but I can't find it now). The idea was that someone trained a neural network to look at live video feeds of passengers moving through a subway station, and control the station's ventilation system. Unfortunately, the movements of individual people were fairly random, whereas the large-scale traffic patterns were extremely regular and periodic. So instead of basing its output on the actual crowd patterns, the neural net decided it was more accurate to look at the hands of an analog clock that happened to be visible through one of its cameras.

All well and good, until the clock stopped working during rush hour, and people started asphyxiating.

Re: The Neural Net Tank Urban Legend

#29

Post by ml_thoughts »

kainolophobia wrote: post #6

>I suggest that dataset bias is real but exaggerated by the tank story, giving a misleading indication of risks from deep learning I don't see how this story gives a "misleading" view of deep learning. From my (admittedly limited) experience with self-driving RC cars, this type of mistake is quite easy for a neural net to make while being quite difficult to detect. In our case, after utilizing a visual back-prop meth…

Another more modern and well-documented example of this would seem to occur in a 2015 write-up of the "Right Whale" competition in Kaggle: http://felixlaumon.github.io/2015/01/08/kaggle-right-whale.h...

Contrary to this author's claims, despite using data augmentation and a fancy modern CNN, a neural network trained to identify whales hit a local optimum where it looked at patterns in waves on the water to identify the whale instead of distinctive markings on the whale's body.

I don't buy the "this isn't a problem in real world applications" argument being made in this article.

Re: The Neural Net Tank Urban Legend

#30

Post by Nomentatus »

For those who say we shouldn't pay too much attention to urban legends about neural network failures, here's a real-life example of neural networks translating "inorganic cat litter" as "in organic cat litter" and thereby creating a real-life half-billion-dollar dirty bomb that genuinely exploded.

https://jonathanturley.org/2014/11/21/kitty-litter-dirty-bom...

This is a hasty link, IIRC the error happened when someone read out instructions aloud to someone else took who was taking notes.

Yes, that badly behaving neural network(s) was human, and therefore far more sophisticated than any we can build yet. Which makes the problem worse and more real, not better or less real.

Post reply on HN