If the deep learning network has enough layers, then can't it start incorporating "abstract" ideas common to any learning task? E.g. could we re-use some layers for image/speech recognition & NLP?
The Limitations of Deep Learning
31–40 of 296 posts
Re: The Limitations of Deep Learning
#32Re: The Limitations of Deep Learning
#33Earlier quoted context omitted.
This terminology goes back to McCulloch and Pitts in 1943, who said they were making an analogy or model based on the behavior of biological neurons. https://en.wikipedia.org/wiki/Artificial_neuron#History There are many things that are inexact about this analogy or model, and many of them were known to be inexact in 1943, but that was the direct inspiration. Apparently there are lots of different mathematical models…
turns out it's very hard to model a thing that we don't know how it actually works
To be fair, we do understand how neurons work, at least on a singular level. Perceptrons model that quite well.
Re: The Limitations of Deep Learning
#34Earlier quoted context omitted.
It isn't anthropomorphizing. There are undeniable architectural similarities between ANN's and biological neural networks. We don't understand either very well yet but the parts we do understand have led to a lot of cross pollination. I don't think computational intelligence will ever match biological networks detail by detail due to the different substrates and resource usage tradeoffs, and they don't need to match.…
This is exactly my point - the danger of "anthropomorphization" lies in taking the brain analogy too far. That is, there shouldn't necessarily be a link between research in neuroscience and advances that make deep learning models more accurate. The tasks are completely different (human learning vs. minimizing a loss function), and it's important for researchers in both fields - neuroscience and AI - to keep that in m…
However, there definitely are analogies! E.g. early work in convnets was inspired by the architecture of cat brains.
I think the fields have useful things to say to each other, but we're getting over a (maybe justified) taboo in talking about machine learning methods being biologically inspired.
Re: The Limitations of Deep Learning
#35As someone primarily interested in interpretation of deep models, I strongly resonate with this warning against anthropomorphization of neural networks. Deep learning isn't special; deep models tend to be more accurate than other methods, but fundamentally they aren't much closer to working like the human brain than e.g. gradient boosting models. I think a lot of the issue stems from layman explanations of neural net…
I don't have ML or deep learning background (no Masters or PhD), adding comment from experience with backtesting trading systems. We will collect market data and design algorithms that seem to produce the kind of outcomes we want. Then test on some other data sets which the algorithms have never been applied on. Many iterations later, you can get a decent profitable algorithm. And if the 'holy grail' algo is run in m…
The "creative" moves may very well come from the search part of the AlphaGo algorithm, though of course the networks have done their jobs of pruning the search space.
Re: The Limitations of Deep Learning
#36I'm sorry, but I don't understand why wider & deeper networks won't do the job. If it took "sufficiently large" networks and "sufficiently many" examples, I don't understand why it wouldn't just take another order of magnitude of "sufficiency."
If you look at the example with the blue dots on the bottom, would it not just take many more blue dots to fill in what the neural network doesn't know? I understand that adding more blue dots isn't easy - we'll need a huge amount of training data, and huge amounts of compute to follow; but if increasing the scale is what got these to work in the first place, I don't see we shouldn't try to scale it up even more.
Re: The Limitations of Deep Learning
#37"Here's what you should remember: the only real success of deep learning so far has been the ability to map space X to space Y using a continuous geometric transform, given large amounts of human-annotated data."
This statement has a few problems - there is no real reason to interpret the transforms as geometric (they are fundamentally just processing a bunch of numbers into other numbers, in what sense is this geometric), and the focus on human-annotated data is not quite right (Deep RL and other things such as representation learning have also achieved impressive results in Deep Learning). More importantly, saying " a deep learning model is "just" a chain of simple, continuous geometric transformations " is pretty misleading; things like the Neural Turing Machine have shown that enough composed simple functions can do pretty surprisingly complex stuff. It's good to point out that most of deep learning is just fancy input->output mappings, but I feel like this post somewhat overstates the limitations.
Re: The Limitations of Deep Learning
#38This is why I don't know if it will be possible (at current limitations) to let insect like brains to fully drive our cars. It may never be good enough.
Re: The Limitations of Deep Learning
#39There is some good information in there and I agree with the limitations he states, but his conclusion is completely made up. "To lift some of these limitations and start competing with human brains, we need to move away from straightforward input-to-output mappings, and on to reasoning and abstraction." There are tens of thousands of scientists and researchers who are studying the brain from every level and we are m…
>I wish we had a faster way to test out more competing theories and models.
Luckily, the state of actual cognitive science and neuroscience is fairly far ahead of, "Gosh there's all these things and we just don't know." Unfortunately, MIT-style cogsci hasn't generated New Jerseyan fast-though-wrong algorithms for Silicon Valley to hype up, so the popular press keeps spreading the myth of our total ignorance.
Besides which, we do know what's missing from deep learning: the ability to express anything other than a trivial Euclidean-space topological structure. We know that real data is sampled from a world subject to cause-and-effect, and that any manifold describing the data should carry the causal structure in its own topology.
Re: The Limitations of Deep Learning
#40>But what if we could learn any program, in a modular and reusable way? Let's see in the next post what the road ahead may look like.
I'm really looking forward to this. If it comes out looking like something faster and more usable than Bayesian program induction, RNNs, neural Turing Machines, or Solomonoff Induction, we'll have something really revolutionary on our hands!