The Limitations of Deep Learning • Hacker News Board

Re: The Limitations of Deep Learning

#111

Post by ndh2 » Mon, Jul 17, 2017, 9:03 PM UTC

zfrenchee wrote: post #70

My qualm with this article is disappointingly poorly backed up. The author makes claims, but does not justify those claims well enough to convince anyone but people who already agree with him. In that sense, this piece is an opinion piece, masquerading as a science. > This is because a deep learning model is "just" a chain of simple, continuous geometric transformations mapping one vector space into another. All it c…

For one, input and output size has to be fixed. All these NNs doing image transformations or recognition only work on fixed-size images. How would you sort a set of integers of arbitrary size using a neural network? What does "solve with a NN" even mean in that context?

Another problems/limitation I can think of is that in NNs you don't have state. The NN can't push something on a stack, and then iterate. How do you divide and conquer using NNs?

Are NNs Turing complete? I don't see how they possibly could be.

Re: The Limitations of Deep Learning

#112

Post by reckoner2 » Mon, Jul 17, 2017, 9:05 PM UTC

ThomPete wrote: post #101

therajiv wrote: post #5

As someone primarily interested in interpretation of deep models, I strongly resonate with this warning against anthropomorphization of neural networks. Deep learning isn't special; deep models tend to be more accurate than other methods, but fundamentally they aren't much closer to working like the human brain than e.g. gradient boosting models. I think a lot of the issue stems from layman explanations of neural net…

One can say that the human mind consist of millions of not very special parts. It's the aggregate, the complexity of which they interact that makes it special. Once you start to connect all these seemingly non-special abilities in deep learning the "magic" starts to happen. You get something that is more than the sum of it's parts. Of course it's not DL in itself thats interesting but the potential emergent complex r…

Has anyone been able to do this? Is anyone working on it?

I only follow the field as a hobby, but as far as I can tell we are nowhere near getting to this point. I think the ability to combine all these parts in a way that the sum is greater than it's parts is going to require many many breakthroughs still.

Re: The Limitations of Deep Learning

#113

Post by pron » Mon, Jul 17, 2017, 9:05 PM UTC

therajiv wrote: post #5

As someone primarily interested in interpretation of deep models, I strongly resonate with this warning against anthropomorphization of neural networks. Deep learning isn't special; deep models tend to be more accurate than other methods, but fundamentally they aren't much closer to working like the human brain than e.g. gradient boosting models. I think a lot of the issue stems from layman explanations of neural net…

One of the greatest clear and present dangers of AI is that various existing algorithms are called just that, rather than what they are: statistical analysis algorithms, or, in short, statistics. Statistics used to be what we called the worst kind of lie; now it's becoming associated with intelligence, hinting at the ability to expose some great hidden truth. The problem lies not only with the algorithms, but with the models they learn (which are indirectly shaped by the algorithms' limitations) that are simplistic to begin with. E.g., they are trained to predict behavior based on a snapshot of statistical data, using either a constant model (which assumes behavior doesn't change over time) or some simplistic first-order model of change. They certainly aren't usually trained to take into account long-term changes or how their own recommendations impact behavior. The result is a powerful yet completely unjustified boost to the public image of statistical data with simplistic change models.

Re: The Limitations of Deep Learning

#114

Post by eli_gottlieb » Mon, Jul 17, 2017, 9:10 PM UTC

CountSessine wrote: post #73

eli_gottlieb wrote: post #63

Earlier quoted context omitted.

>In particular, I'm thinking of Dual Process Theory Which has been at least partly debunked as psychology's replication crisis went on, and has been called into question on the neuroscientific angle as well.

Partly, yes - especially with ego depletion on the ropes. I'm not sure that dual process theory needs to be thrown out along with ego depletion, though.

I can see three reasons to "throw it out":

1) Replication failure, plain and simple.

2) Overfitting. There are dozens to hundreds of "cognitive biases" on lists: https://en.wikipedia.org/wiki/List_of_cognitive_biases. When you have hundreds of individual points, you really ought to draw some principles, and the principle should not be, "The system generating all this is rigid and inflexible."

3) Imprecision! Again, dozens to hundreds of cognitive biases. What possible behavior or cognitive performance can't be assimilated into the heuristics and biases theory? What can falsify it overall, even after so many of its individual supporting experiments and predictions have fallen down?

It looks like a mere taxonomy of observations, not a substantive theory.

Re: The Limitations of Deep Learning

#115

Post by reckoner2 » Mon, Jul 17, 2017, 9:12 PM UTC

kowdermeister wrote: post #106

> In short, deep learning models do not have any understanding of their input, at least not in any human sense. Our own understanding of images, sounds, and language, is grounded in our sensorimotor experience as humans—as embodied earthly creatures. Well maybe we should train systems with all our sensory inputs first, like newborns leans about the world. Then make these models available open source like we release o…

This approach has always interested me. I can train an decent Cats Vs. Dogs classifier in a few minutes. But real human intelligence takes many years of continuous and varied input to develop.

Are there systems out there that are taking influence from newborns being exposed to the world? An unsupervised learning system with a huge array of inputs running for years?

Re: The Limitations of Deep Learning

#116

Post by LeanderK » Mon, Jul 17, 2017, 9:17 PM UTC

the author raises some valid points, but i don't like the style it is written in. He just makes some elaborate claims about the limitation of Deep Learning, but conveys why they are limitations. I don't disagree about the fact that there are limits to Deep Learning and many may be impossible to overcome without completely new approaches. I would like to see more emphasis on why things, like generating code from descriptions, that are theoretically possible, are absolutely impossible and out of reach today and not make the intention that the tasks itself is impossible (like the halting-problem).

Re: The Limitations of Deep Learning

#117

Post by SomeStupidPoint » Mon, Jul 17, 2017, 9:20 PM UTC

kxyvr wrote: post #90

SomeStupidPoint wrote: post #77

Earlier quoted context omitted.

You seem to have replied on a tangent: how is what you describe not just "curve fitting"? Humans didn't magic that model up: you're ignoring the huge amount of human effort over thousands of years that it took to arrive at that model. If we gave a ML algorithm a similar amount of time and asked it to construct a simple model of the situation, it might very well hand back the formula you presented. Your entire post ba…

I believe the process for deriving fundamental physical models differs from the techniques used in ML. For example, say we want to use the principle of least action to derive an expression for energy similar to what Landau and Lifshitz derive in their book Mechanics. Here, we assume that the motion of a particle is defined by its position and velocity. We assume that the motion of the particle is defined by an optimi…

You haven't explained how the first case isn't "curve fitting": the agents performing the compilation of those facts into the new fact are just spitting out the "best" fit string of symbols based on learned rules, etc etc. Somethings computers can (theoretically) do, and which fits the description "curve fitting" just fine. School (and other education) is training the model they're using to do that compilation, but it's still just "curve fitting" based on reward/punishment signals.

What part of that can't an ML agent learn to do?

From my perspective, you're just describing the "higher order" layers of the network and pretending that humans aren't actually running those functions embedded on deep networks, then proclaiming that deep networks can't do it.

Re: The Limitations of Deep Learning

#118

Post by DiThi » Mon, Jul 17, 2017, 9:29 PM UTC

reckoner2 wrote: post #112

ThomPete wrote: post #101

Earlier quoted context omitted.

One can say that the human mind consist of millions of not very special parts. It's the aggregate, the complexity of which they interact that makes it special. Once you start to connect all these seemingly non-special abilities in deep learning the "magic" starts to happen. You get something that is more than the sum of it's parts. Of course it's not DL in itself thats interesting but the potential emergent complex r…

Has anyone been able to do this? Is anyone working on it? I only follow the field as a hobby, but as far as I can tell we are nowhere near getting to this point. I think the ability to combine all these parts in a way that the sum is greater than it's parts is going to require many many breakthroughs still.

The problem is that we don't really know for sure. We kind of predict things by extrapolating what we know and what we have, but we can never be sure there won't be any sudden breakthroughs.

Re: The Limitations of Deep Learning

#119

Post by Cacti » Mon, Jul 17, 2017, 9:34 PM UTC

eanzenberg wrote: post #3

This point is very well made: 'local generalization vs. extreme generalization.' Advanced NN's today can locally generalize quite well and there's a lot of research spent to inch their generalization further out. This will probably be done by increasing NN size or increasing the NN building-blocks complexity.

I would really like to hear a definition of what generalization means, because I don't think we have one.

Unless we're talking generalization to arbitrary distributions, which is of course unsolvable.

Re: The Limitations of Deep Learning

#120

Post by kowdermeister » Mon, Jul 17, 2017, 9:37 PM UTC

reckoner2 wrote: post #115

kowdermeister wrote: post #106

> In short, deep learning models do not have any understanding of their input, at least not in any human sense. Our own understanding of images, sounds, and language, is grounded in our sensorimotor experience as humans—as embodied earthly creatures. Well maybe we should train systems with all our sensory inputs first, like newborns leans about the world. Then make these models available open source like we release o…

This approach has always interested me. I can train an decent Cats Vs. Dogs classifier in a few minutes. But real human intelligence takes many years of continuous and varied input to develop. Are there systems out there that are taking influence from newborns being exposed to the world? An unsupervised learning system with a huge array of inputs running for years?

All these current examples of AI and ML are just a very small fraction of what we mean by intelligence so I'm not surprised by the pessimistic posts that hit HN from time to time.

Training systems with rich real world experiences sounds something that Open AI should be developing. It's probably not something that you can do over a weekend plus it takes serious of funding and wetware so it's probably the reason it's not there yet.