Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Limitations of Deep Learning

blog.keras.io

Post reply on HN
Print view View original

281–290 of 296 posts

Re: The Limitations of Deep Learning

#281

Post by nojvek »

latently wrote: post #136
geofft wrote: post #89

Earlier quoted context omitted.

That's about as interesting as saying that a Taylor series can approximate any analytic function arbitrarily well given time and space. Or that a lookup table can approximate any function arbitrarily well given time and space: see also the Chinese room example. The first question is whether that neural network is learnable . Sure, some configuration of neurons may exist. Is it possible given enough time and space to…

I like your comment. The real question is whether they are conscious. The analogy between deep neural networks and the brain has proven to be very fruitful. Other analogies may as well. See our upcoming paper for more info. https://grey.colorado.edu/mediawiki/sites/mingus/images/3/3a...

I think a lot of people end up mixing being alive with being conscious. Is a tree conscious? Is a self driving car conscious?

If we use the definition "Aware of its surroundings, responding and acting towards a certain goal" then a lot of things fit that definition.

When an AI plays the atari games, learns from it and plays at a human level, I would call it conscious. It's not a human level conscious agent but conscious nonetheless.

Re: The Limitations of Deep Learning

#282

Post by randcraw »

webmaven wrote: post #165
cm2187 wrote: post #133

Earlier quoted context omitted.

well, you are mentioning an example where: - there is data - there is a wide market that could justify large investments in AI With this combination, yeah I can see AI being used. In fact medecine is one of the few professions that never industrialised. But there are loads of other professions where either or none of the conditions above are met. If you are talking about a doctor specialised in a rare disease, where…

> If you are talking about a doctor specialised in a rare disease, where there is very little data, and very few patients to cure, how do you think AI will replace that? Well, since transfer learning is a thing, you would start with a general purpose medical system and then train it on what little data you do have on the rare disease to produce an appropriate model (which isn't too different from the way a human expe…

Yes, experts will use AI to build better tools that will be accessible via computer and by less-expert physicians to augment their skill base in diagnosis. Trying to fully automate the physician promises to be so difficult and expensive (and legally fraught) that the value it adds will be nowhere near worth the investment. No doubt a few direct-to-patient AI-based apps and services will arise in this space, and maybe AI will allow generalists will extend their reach further into the space of specialists. But robot doctors will remain the stuff of fantasy for many decades yet, I suspect.

Re: The Limitations of Deep Learning

#284

Post by randcraw »

nv-vn wrote: post #173
meh2frdf wrote: post #128

Correct me if I'm wrong but I don't see that with 'deep learning' we have answered/solved any of the philosophical problems of AI that existed 25 years ago (stopped paying attention about then). Yes we have engineered better NN implementations and have more compute power, and thus can solve a broader set of engineering problems with this tool, but is that it?

Yep. The whole machine learning craze is just fueled by the fact that it's now feasible to create models for handwriting/voice/image recognition that actually work reliably. But in terms of the underlying technology, we haven't had some "breakthrough" that explained how the brain works or anything even close to that.

I 90% agree, but if that were all there was to it, we should have been able to achieve the same level of success decades earlier, just by running our ML code longer. Given enough time, old slow AI software should have produced the same analytical results as deep learning does today.

But that's not the case. Deep nets can model vastly more information / state than any other AI/ML method. Once Hinton (and others) showed how to train NNs with more than three layers (ca. 2006) it was finally possible to learn and store all that state. Then with the rise of GPGPUs soon after, deep nets became efficient as well. Thereafter several tasks that had been infeasible even using curated information became amenable to mostly brute force learning strategies driven only by labeled examples -- just lots of 'em.

The question now is how far can we extend DL's tools and examples. Are they sufficient to build higher level cognitive AI agents. Must AGI employ many thousands of deep nets? Or can all those specific-skill nets be folded together somehow into one unified "deep mind"?

Like you, I'm doubtful that today's very specific successes in DL will lead to higher level cognition in the foreseeable future. That path isn't at all clear to me.

Re: The Limitations of Deep Learning

#285

Post by randcraw »

jonobelotti wrote: post #152
kowdermeister wrote: post #106

> In short, deep learning models do not have any understanding of their input, at least not in any human sense. Our own understanding of images, sounds, and language, is grounded in our sensorimotor experience as humans—as embodied earthly creatures. Well maybe we should train systems with all our sensory inputs first, like newborns leans about the world. Then make these models available open source like we release o…

Brooks' 'Intelligence Without Representation' ( http://people.csail.mit.edu/brooks/papers/representation.pdf ) starts with a pretty strong argument imo against the story of 'stick-together' AGI you're describing.

I think Brooks' Cog initiative was an attempt to 'ground' the robot's perceptions of the physical world into forming a rich scalable representation model. But it looks like that line of investigation ended ~2003 with Brooks' retirement. Too bad, given the seeming suitability of using deep nets to implement it.

http://www.ai.mit.edu/projects/humanoid-robotics-group/cog/o...

Re: The Limitations of Deep Learning

#286

Post by randcraw »

nshm wrote: post #62
gallerdude wrote: post #36

I'm sorry, but I don't understand why wider & deeper networks won't do the job. If it took "sufficiently large" networks and "sufficiently many" examples, I don't understand why it wouldn't just take another order of magnitude of "sufficiency." If you look at the example with the blue dots on the bottom, would it not just take many more blue dots to fill in what the neural network doesn't know? I understand that addi…

"sufficiently large" could be much more than number of atoms in the universe. You just do not have resources to run computation at such scale.

This is my problem with the thesis that simply scaling deep nets to new heights will ultimately subsume all brain function. If it takes weeks to train a simple object recognizer deep net, how long would it take a grand unified deep net to learn to tie its shoelaces? Puberty?

Re: The Limitations of Deep Learning

#287

Post by nerdponx »

Gtifn wrote: post #138
kxyvr wrote: post #131

Earlier quoted context omitted.

Alright, so from my perspective, curve fitting consists of three things 1. Definition of a model. ML models like multilayer perceptrons used a superposition of sigmoids, but newer models have superpositions of other functions and more nested hierarchies. 2. A metric to define misfit. Most of the time we use least squares because it's differentiable, but other metrics are possible. 3. An optimization algorithm to mini…

What do you think about a bayesian interpretation of the above as MAP/MLE? https://arxiv.org/abs/1706.00473

Interesting abstract. I love Bayesian stats so hopefully this will be a fun commute read. Thanks!

Re: The Limitations of Deep Learning

#288

Post by nerdponx »

urethrafranklin wrote: post #283
kinkrtyavimoodh wrote: post #32

Earlier quoted context omitted.

I am sure he does.

His public statements would indicate otherwise.

Consider that his public statements are made on the advice of his publicist, and that encouraging the AI hype is self-serving.

Re: The Limitations of Deep Learning

#289

Post by kxyvr »

nerdponx wrote: post #254
kxyvr wrote: post #90

Earlier quoted context omitted.

I believe the process for deriving fundamental physical models differs from the techniques used in ML. For example, say we want to use the principle of least action to derive an expression for energy similar to what Landau and Lifshitz derive in their book Mechanics. Here, we assume that the motion of a particle is defined by its position and velocity. We assume that the motion of the particle is defined by an optimi…

Of course. "What do I fit this curve to" is a prerequisite to "what is the shape of this curve?" You shouldn't feel the need to defend theory-based modeling against some imagined incursion from arrogant deep learning researchers. NNs work tremendously well in a few specific problem domains that we had no way to approach otherwise. Elsewhere, they're not much better than any other prediction algorithm. By the way XGBo…

I very much agree! Barring some kind of special intuition to the problem, I think ML are a fantastic tool for building models from empirical data. Even with intuition, sometimes they work as well. My core argument is that anthropomorphizing the algorithms has led to a great deal of confusion as to when we should or should not use these models. I often do computational modeling work with engineers and many of them are starting to eschew good, foundationaly sound models for ML not because they work better, in fact, on many of these problems they work far, far worse, but because good computational modeling is hard and it sounds like all they have to do with ML is teach the algorithms how physics works and how to be an engineer. Since they're good teachers, they should be able to teach the algorithm, right? In reality, it's still dirty, grinding computational modeling work. If we just called these models what they really are, empirical models, I think there'd be far less confusion as to when they should be used.

Re: The Limitations of Deep Learning

#290

Post by backpropaganda »

erikb wrote: post #243
backpropaganda wrote: post #240

Earlier quoted context omitted.

1. Imagine infinite compute capability. Exhaustively play all possible games, and use that to figure out best moves at any state. This is essentially what Alphago did, but using translation variance to reduce the search space. 2. There is no contradiction here. We just have to accept that human-like play can emerge from memorization.

Can you explain what from your point of view is the difference between AlphaGo and a Chess AI? Because to me it sounds like the one should have resulted as an evolution from the other if it would be that simple.

Yes. In Chess, it's relatively easy to judge how good a board position is, and thus people have been successful by hand-engineering the board position evaluator (also called value function in RL lingo), and then just doing tree search to take the action which improves board position the most. In Go, evaluating board position is much more difficult, and it's not possible to approximate the value function by hand-engineered code. Thus, AlphaGO approximates the value by simulating the game till win/lose from arbitrary board positions to evaluate its value. This doesn't really require neural networks. You could also do the same with table lookup. What neural networks offer here is some translation invariance generalization, and capability to compress the table into fewer parameters by identifying common input features. It's possible to achieve AlphaGo performance by just having a BIG table of state-values and using some kernel to do nearest neighbour search (such as done by deepmind here: https://arxiv.org/abs/1606.04460)
Post reply on HN