41–50 of 67 posts
I first encountered this idea in a sci-fi story (I want to say it was one of Peter Watts' "Rifters" novels, but I can't find it now). The idea was that someone trained a neural network to look at live video feeds of passengers moving through a subway station, and control the station's ventilation system. Unfortunately, the movements of individual people were fairly random, whereas the large-scale traffic patterns wer…
1. Training on a biased data set leads to biased predictions. This is undoubtedly true.
2. Data sets can be biased in unexpected and unforeseen ways, so therefore productions can also be biased in unexpected and unforeseen ways. The examples at the end of this article don't quite touch on that point. But examples of this abound in social science. Eg: https://blog.conceptnet.io/2017/07/13/how-to-make-a-racist-a...
3. Deep and convolutional neural networks are susceptible to this phenomenon. This is the point that the article is debating.
More importantly, it's obvious that "it" could definitely happen and in fact happens a lot- "it" being overfitting to examples. Machine learning classifiers suffer from this a lot, it's the whole bias/variance tradeoff issue. Neural Nets are not only not immune to overfitting, they are even particularly vulnerable to it (especially the ones with millions of parameters). We've probably all read the adversarial examples papers- a clear case of overfitting to irrelevant details ("noise").
The story (apocryphal or not) seems like a cautionary tale against overfitting, or a not-so-innocent attempt to poke fun at machine learning researchers. One way or another, overfitting is no joke and it's definitely no urban legend.
For a better, actual example of this problem, see the leopard sofa: http://rocknrollnerd.github.io/ml/2015/05/27/leopard-sofa.ht...
Rainer Kordmaa • 6 months ago
Kinda reminds me a story of how a neural network was
trained by military to detect camouflaged tanks on
terrain, except pictures with tanks were taken on a nice
sunny day and pictures without on a cloudy day and instead
of a tank detector they ended up with a sunny day detectorI get the author's feelings about why we shouldn't tell this story, but I still disagree. It's a pithy, funny example of GIGO in machine learning. People could read conclusions about the abilities of neural networks from the story, but they're wrong to do so -- it's a PEBKAC error, not a technology one. "Truthy" cautionary tales are a near-universal feature of human cultures -- why shouldn't machine learning have som…
It's a plausible story. Fine to present as a parable, but we should stop presenting it as true unless we can find a reliable source for it. Until today, I believed it was true. It was told to me as an undergrad, by a professor who believed it himself.
Many times it seems like people go into these things hoping that the machine learning part will figure out things for them and relieve themselves of the problem of thinking hard about the problem. It doesn't. It only moves your problem over a bit and increases the difficulty.
In fact this problem pops up even in pedagogy where the lessons people are taught actually train them to do the wrong thing (for example pilots responding to aircraft attitude upsets).
The parable's lesson is a simplistic one, basically: "stop and think about what you're doing". But like other simple lessons about crying wolf or stitching in-time, it bears repeating.
Earlier quoted context omitted.
It's a plausible story. Fine to present as a parable, but we should stop presenting it as true unless we can find a reliable source for it. Until today, I believed it was true. It was told to me as an undergrad, by a professor who believed it himself.
I first heard it in an AI class as well. When I relate the story I usually give it as an anecdote of apocryphal origin. But I do love the story, because it brings into focus several issues: what do you actually want your classifier to learn? what does your training set actually teach? how do you know you've learned the right thing? Many times it seems like people go into these things hoping that the machine learning…
Well, that's not quite true. In robot sensing several things have recently moved from the nigh-on-impossible column to the holy-shit-that-actually-works-pretty-well column, thanks to ML.
But I agree with the rest of it.
Scare-quotes dataset bias (in science we say "selection bias") is the bread and butter of any field that doesn´t get an opportunity to fine-tune sample design issues. There's even hierarchical models with an equation giving the probability that an item will be observed at all, conditioned to known features. Those who don't know their statistical models are bound to reinvent statistical theory.
From this viewpoint, I found the section where the author lengthly argues how this could not possibly happen with the current state of the art visual task CNNs (especially because people apply preprocessing steps such as whitening and augmentation to get rid of exactly this kind of biases), let's say, weird. The parable is not about CNNs, it is about the importance of paying attention what features your model will extract from the training dataset and whether your model is learning the right things.
The whole "Could it happen"? section is a bit strange. On the one hand, it focuses on CNNs when it's clear we're talking about a binary classifier (the article itself points that out). If Fredkin was really the originator of the story, then discussing CNNs is an anachronism (they were 30 years away at the time). More importantly, it's obvious that "it" could definitely happen and in fact happens a lot- "it" being ove…
Can you give an example of where overfitting happened and was successfully corrected for?