I don't get it. If reasoning is not an option how does deep learning beat the boardgame go?
Memorisation + small amounts of generalization.
The Limitations of Deep Learning
161–170 of 296 posts
Re: The Limitations of Deep Learning
#162Earlier quoted context omitted.
> The quality of the algo and I assume the deep learning model lies in the quality (breadth and depth) of the data, and how honest with himself the person choose to model it. I've only dabbled with machine-learning here and there for the past 10 years or so, but if there's one thing I've learned so far is that the data behind your ML code (and the way it is structured) is responsible for almost all the success or fai…
> focus a lot of his time on the data itself... from where he intends to collect it? how is it structured? is it reliable? is it "enough"? What's the best books on this subject? I suppose it's a very broad topic and thus more difficult to talk about than a single "neural network" algorithm.
Interested in what part of that you feel needs to be explained in more depth? Not sure reading several books is necessary for explaining data collection and data munging...to me it's definitely something best learned by doing.
work in data analysis/stats
Re: The Limitations of Deep Learning
#163Earlier quoted context omitted.
This is exactly my point - the danger of "anthropomorphization" lies in taking the brain analogy too far. That is, there shouldn't necessarily be a link between research in neuroscience and advances that make deep learning models more accurate. The tasks are completely different (human learning vs. minimizing a loss function), and it's important for researchers in both fields - neuroscience and AI - to keep that in m…
How do you know your brain's not minimizing a loss function?
For the analogy to hold, it's more of a question of whether or not ML algorithms operate in the same way as the brain. Right now, ML models use algorithms from continuous optimization that require certain structure. Namely, we require a Hilbert space, so that we can define things like derivatives and gradients. This puts certain requirements on the kinds of functions that we can minimize and the kinds of spaces that we can work with. These are requirements that are difficult to have precise analogies in biology. What does it mean to have an inner product in the brain? We does twice continuously differentiable mean in the context of a neuron? Even if there is a minimization principle, which I am not sure there is or is not, if ML uses algorithms, which are fundamentally not realizable in biology, how can we say it replicates the brain?
Re: The Limitations of Deep Learning
#164As someone primarily interested in interpretation of deep models, I strongly resonate with this warning against anthropomorphization of neural networks. Deep learning isn't special; deep models tend to be more accurate than other methods, but fundamentally they aren't much closer to working like the human brain than e.g. gradient boosting models. I think a lot of the issue stems from layman explanations of neural net…
Well said. It's just curve fitting.
Or finding eigenvalues.
Re: The Limitations of Deep Learning
#165Earlier quoted context omitted.
I'd say on the contrary, the problem with experts is that they are so expensive to train and so rare. It is easier to collect data, train the AI and then equip doctors all over the world with it than to have thousands of experts in that particular field. A doctor that treats patients all day long doesn't have time to keep up with the research and state of the art. A researcher that is on the cutting edge of medicine…
well, you are mentioning an example where: - there is data - there is a wide market that could justify large investments in AI With this combination, yeah I can see AI being used. In fact medecine is one of the few professions that never industrialised. But there are loads of other professions where either or none of the conditions above are met. If you are talking about a doctor specialised in a rare disease, where…
> If you are talking about a doctor specialised in a rare disease, where there is very little data, and very few patients to cure, how do you think AI will replace that?
Well, since transfer learning is a thing, you would start with a general purpose medical system and then train it on what little data you do have on the rare disease to produce an appropriate model (which isn't too different from the way a human expert is produced). In fact, I would assume that the first such systems will be created and used by the researchers focusing on rare diseases.
Re: The Limitations of Deep Learning
#166As someone primarily interested in interpretation of deep models, I strongly resonate with this warning against anthropomorphization of neural networks. Deep learning isn't special; deep models tend to be more accurate than other methods, but fundamentally they aren't much closer to working like the human brain than e.g. gradient boosting models. I think a lot of the issue stems from layman explanations of neural net…
> Pretty much every time DL is covered by media, there has to be some contrived comparison to human brains Well, what we've done so far is emulate maybe 1 mm^3 of brain matter - some isolated, very specialized functional blocks in the greater architecture of the brain. They behave as expected - are experts on very narrow topics, but of course fail to integrate their functioning with a larger body of knowledge, becaus…
> Well, what we've done so far is emulate maybe 1 mm^3 of brain matter - some isolated, very specialized functional blocks in the greater architecture of the brain. They behave as expected - are experts on very narrow topics, but of course fail to integrate their functioning with a larger body of knowledge, because that body just isn't there (yet).
I think you're falling into the same anthropomorphism trap that the GP is talking about. We haven't even breached the most important topic: neural plasticity - a brain's ability to rewire itself based on a complex feedback loop driven by environmental inputs (which are, at this point in human development, an almost infinitely more complex system of culture built up over tens of thousands of years). From my work in neuroscience, it seems that the computational complexity of the state of the art DL algorithms barely register when compared to a network of a few hundred biological neurons like the nervous system of Caenorhabditis elegans, which is itself far less capable of self reorganization than even the simplest mammalian brain. Hell, even the most basic potentiation that you'd find in decades old research on addiction is far outside the scope of modern machine learning research and we don't yet have any clean mathematical theories that can emulate plasticity like back propagation or gradient descent can with simple learning.
The current hype around neural networks is the equivalent of saying that we've analytically solved the n-body problem when all we've done is solve a system of equations with two linear variables. The domains are connected but only in the trivial sense that both have variables named "x" and "y."
Re: The Limitations of Deep Learning
#167> In short, deep learning models do not have any understanding of their input, at least not in any human sense. Our own understanding of images, sounds, and language, is grounded in our sensorimotor experience as humans—as embodied earthly creatures. Well maybe we should train systems with all our sensory inputs first, like newborns leans about the world. Then make these models available open source like we release o…
Brooks' 'Intelligence Without Representation' ( http://people.csail.mit.edu/brooks/papers/representation.pdf ) starts with a pretty strong argument imo against the story of 'stick-together' AGI you're describing.
Thanks for the link to this interesting paper.
I think we're seeing some recapitulation of those arguments WRT 'ensembles of DL models' approaches.
Re: The Limitations of Deep Learning
#168If the deep learning network has enough layers, then can't it start incorporating "abstract" ideas common to any learning task? E.g. could we re-use some layers for image/speech recognition & NLP?
this is exactly what happens in transfer learning. A recent paper by google ( https://research.googleblog.com/2017/07/revisiting-unreasona... ) shows that pre-training on a very large image database leads to improvements in state of the art for several different image problems. This is because the weights required for one image problem are not necessarily all that different from another image problem, especially in t…
Transfer learning has been shown to improve training times in other modes (such using an image classification model to initialize an NLP model) over randomly initialized values.
Re: The Limitations of Deep Learning
#169Earlier quoted context omitted.
> focus a lot of his time on the data itself... from where he intends to collect it? how is it structured? is it reliable? is it "enough"? What's the best books on this subject? I suppose it's a very broad topic and thus more difficult to talk about than a single "neural network" algorithm.
Interested in what part of that you feel needs to be explained in more depth? Not sure reading several books is necessary for explaining data collection and data munging...to me it's definitely something best learned by doing. work in data analysis/stats
öür bëst frïënd's sïstër ġëts 74 än höür ön thë ïntërnët, shë's bëën öüt öf wörk för 8 mönths, thë prëvïöüs mönth hër ċhëċk wäs 20166 önlÿ wörkïnġ ön thë ïntërnët 4 höürs ëvërÿ däÿ... sëë thïs lïnk
ＡＭＡZＩＮＧ ＪＯＢＳRe: The Limitations of Deep Learning
#170Then people are assuming Deep Learning can be applied to a Self Driving Car System end-to-end! Can you imagine the outcome?!
Yes. Death Race 2000.