Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Limitations of Deep Learning

blog.keras.io

Post reply on HN
Print view View original

151–160 of 296 posts

Re: The Limitations of Deep Learning

#151

Post by bpicolo »

goatlover wrote: post #122
sadlion wrote: post #104

Earlier quoted context omitted.

I sincerely would like to know what you think the alternatives are?

Sounds something like Haskell with a Smalltalk environment. Functional, statically typed with powerful type extensions, but with an image instead of text files that you modify. From just using Jupyter Notebooks, I can see the appeal of working with a live environment, and it's just a fancy REPL, not a full Lisp or Smalltalk environment.

Programming is describing a solution space, and we describe things with words. I don't see how anything but text/speech would map to that aspect of programming.

Re: The Limitations of Deep Learning

#152

Post by jonobelotti »

kowdermeister wrote: post #106

> In short, deep learning models do not have any understanding of their input, at least not in any human sense. Our own understanding of images, sounds, and language, is grounded in our sensorimotor experience as humans—as embodied earthly creatures. Well maybe we should train systems with all our sensory inputs first, like newborns leans about the world. Then make these models available open source like we release o…

Brooks' 'Intelligence Without Representation' (http://people.csail.mit.edu/brooks/papers/representation.pdf) starts with a pretty strong argument imo against the story of 'stick-together' AGI you're describing.

Re: The Limitations of Deep Learning

#153

Post by kxyvr »

Gtifn wrote: post #138
kxyvr wrote: post #131

Earlier quoted context omitted.

Alright, so from my perspective, curve fitting consists of three things 1. Definition of a model. ML models like multilayer perceptrons used a superposition of sigmoids, but newer models have superpositions of other functions and more nested hierarchies. 2. A metric to define misfit. Most of the time we use least squares because it's differentiable, but other metrics are possible. 3. An optimization algorithm to mini…

What do you think about a bayesian interpretation of the above as MAP/MLE? https://arxiv.org/abs/1706.00473

Unless I'm missing something, and I likely am, the linked paper is still based on the the fundamental assumptions behind curve fitting that I listed above. Namely, their optimization algorithms, metrics, and models are still based on Hilbert spaces even though they've added stochastic elements and more sophisticated models.

Re: The Limitations of Deep Learning

#154

Post by Cybiote »

sqeaky wrote: post #129
s-macke wrote: post #108

Earlier quoted context omitted.

An ANN still resembles major features of an bio-NN. 1. A network 2. Flow of information is mainly unidirectional through a node 3. Multiple inputs, but one output, which is connected to the inputs of other neurons. 4. The connection strength between 2 neurons can be changed. 5. Non-linear behavior. After all, I think, this is not such a bad first approximation. Hence the picture in the middle. But I cannot believe th…

When there is feedback in neurons, what do you think that conveys? I agree it is not some simple error correction like what is propagated backwards, but it happens often and I presume its something useful or it wouldn't be there.

Top down predictions are likely mediated by feedback connections from higher to lower areas. Functions include possibly encoding a generative prior for prediction, speeding up inference. They also play an important role in coding more informative error signals than simple derivatives and are part of how the brain learns even as it predicts.

Re: The Limitations of Deep Learning

#155

Post by webmaven »

philipkglass wrote: post #94
robotresearcher wrote: post #84

Earlier quoted context omitted.

McCulloch's argument was that perhaps the gross behaviour of a NN as layers of simple transfer functions is where the real action is, and the rest of the details are just gravy. The fact we now give this to undergrads as homework suggests that there was some value to this idea.

Students in computer science may implement a perceptron as a homework problem. Students in biology don't do that, nor do they use perceptrons to learn about brains, because perceptrons bear only faint resemblance to biological neurons. Reproducing important biological features of real neurons requires much more complicated software. I'm not denigrating perceptrons or other neuro-inspired approaches to classification.…

But it turns out that they don't have to be. We know that radically different low-level implementations can approximate the same higher-level functions given a large enough network and enough training (eg. half-precision floating point, integer, or even binary ANNs, not to mention the wide variety of activation functions such as relu, sigmoid, tanh, maxout, softmax, etc.), and we've seen increasingly varied ANN architectures applied to the same tasks with good results, so I would expect this to continue to hold true for ever more sophisticated tasks.

I am certain, BTW, that further study of biological neurons will continue to yield insights for the design of ANNs, but it does not at all follow that ANN design will become more similar to biological NNs as a result. Given the completely different substrates, simulating a biologically plausible NN in order to perform a task (for purposes other than gaining further understanding of biological NNs, that is) would be incredibly wasteful and unnecessary, even if your goal is to create an AGI of some sort.

Re: The Limitations of Deep Learning

#156

Post by jd007 »

Cacti wrote: post #119
eanzenberg wrote: post #3

This point is very well made: 'local generalization vs. extreme generalization.' Advanced NN's today can locally generalize quite well and there's a lot of research spent to inch their generalization further out. This will probably be done by increasing NN size or increasing the NN building-blocks complexity.

I would really like to hear a definition of what generalization means, because I don't think we have one. Unless we're talking generalization to arbitrary distributions, which is of course unsolvable.

I tend to think of "generalization", in the general sense (pun not intended), to be information compression.

We are simply trying to answer the question: what is the shortest description (i.e. most informationally compressed/dense version) that fits what we see in this infinite (at least to us mortals) universe of ours? Mathematically the length of such a description can be thought of as the Kolmogorov complexity.

Edit: I should add, the information compression performed when generalizing can (and often is) lossy.

Re: The Limitations of Deep Learning

#157

Post by Barbara454 »

yters wrote: post #92
paganel wrote: post #58

Earlier quoted context omitted.

> The quality of the algo and I assume the deep learning model lies in the quality (breadth and depth) of the data, and how honest with himself the person choose to model it. I've only dabbled with machine-learning here and there for the past 10 years or so, but if there's one thing I've learned so far is that the data behind your ML code (and the way it is structured) is responsible for almost all the success or fai…

The data processing inequality says processing data does not increase its information content.

I just got paid $6784 working off my laptop this month. And if you think that's cool, my divorced friend has twin toddlers and made over $9k her first month. It feels so good making so much money when other people have to work for so much less. This is what I do,==http://bit.do/dnRs5

Re: The Limitations of Deep Learning

#158

Post by nojvek »

raverbashing wrote: post #126
romaniv wrote: post #87

Earlier quoted context omitted.

I think it would help a lot if we brought random forests and SVMs to the same level of performance as DNNs. Demonstrating that more "mechanical" algorithms can be as efficient would dispel some of the anthropomorphism and allow for better analysis of why certain things work. I also believe that researches have responsibility to outline the limits of their own algorithms in research papers. (For example, presenting ex…

Not possible, unfortunately

I've occasionally found that SVM's work great for one shot learning if you have good features and nicely labelled dataset. CNN's are really good at extracting features. Once you've extracted features that are generic, using an SVM as the last layer to train while keeping the CNN parameters intact yields great accuracy.

I think that's where we are really headed. A combination of deep learning, boosted trees, svm, evolutionary algos, knowledge graphs e.t.c all stitched together to build stronger AI systems.

Remember our aeroplanes don't flap wings but still carry tonnes of weight and fly half way around the world. Once we discovered fundamentals of aerodynamics a lot of supernatural things were possible.

Same with intelligence, once we discover the essentials of intelligence and mathematically formulate it, supernatural intelligence is very possible. This is the thing that really scares people. I have no idea how close we are to it, but I'm sure it will change society the way internet and mobile phones changed the world.

Re: The Limitations of Deep Learning

#159

Post by ilaksh »

Actually there are quite a few researchers working on applying newer NN research to systems that incorporate sensorimotor input, experience, etc. and more generally, some of them are combining an AGI approach with those new NN techniques. And there has been research coming out with different types of NNs and ways to address problems like overfitting or slow learning/requiring huge datasets, etc. When he says something about abstraction and reasoning, yes that is important but it seems like something NNish may be a necessary part of that because the logical/symbolic approaches to things like reasoning have previously mainly been proven inadequate for real-world complexity and generally the expectations we have for these systems.

Search for things like "Towards Deep Developmental Learning" or "Overcoming catastrophic forgetting in neural networks" or "Feynman Universal Dynamical" or "Wang Emotional NARS". No one seems to have put together everything or totally solved all of the problems but there are lots of exciting developments in the direction of animal/human-like intelligence, with advanced NNs seeming to be an important part (although not necessarily in their most common form, or the only possible approach).

Re: The Limitations of Deep Learning

#160

Post by backpropaganda »

andreyk wrote: post #37

"Here's what you should remember: the only real success of deep learning so far has been the ability to map space X to space Y using a continuous geometric transform, given large amounts of human-annotated data." This statement has a few problems - there is no real reason to interpret the transforms as geometric (they are fundamentally just processing a bunch of numbers into other numbers, in what sense is this geome…

Just because there's a paper on it, and the model has a name, doesn't mean it works. NTM and deep RL don't work for real problems.
Post reply on HN