Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Limitations of Deep Learning

blog.keras.io

Post reply on HN
Print view View original

101–110 of 296 posts

Re: The Limitations of Deep Learning

#101

Post by ThomPete »

therajiv wrote: post #5

As someone primarily interested in interpretation of deep models, I strongly resonate with this warning against anthropomorphization of neural networks. Deep learning isn't special; deep models tend to be more accurate than other methods, but fundamentally they aren't much closer to working like the human brain than e.g. gradient boosting models. I think a lot of the issue stems from layman explanations of neural net…

One can say that the human mind consist of millions of not very special parts. It's the aggregate, the complexity of which they interact that makes it special.

Once you start to connect all these seemingly non-special abilities in deep learning the "magic" starts to happen. You get something that is more than the sum of it's parts. Of course it's not DL in itself thats interesting but the potential emergent complex relationships.

Re: The Limitations of Deep Learning

#102

Post by cs702 »

Yes.

Here's how I've been explaining this to non-technical people lately:

"We do not have intelligent machines that can reason. They don't exist yet. What we have today is machines that can learn to recognize patterns at higher levels of abstraction. For example, for imagine recognition, we have machines that can learn to recognize patterns at the level of pixels as well as at the level of textures, shapes, and objects."

If anyone has a better way of explaining deep learning to non-technical people in a few short sentences, I'd love to see it. Post it here!

Re: The Limitations of Deep Learning

#103

Post by visarga »

cm2187 wrote: post #42

I think the requirement for a large amount of data is the biggest objection to the reflex "AI will replace [insert your profession here] soon" that many techies, in particular on HN, have. There are many professions where there is very little data available to learn from. In some case (self-driving), companies will invest large amount of money to build this data, by running lots of test self-driving cars, or paying p…

I'd say on the contrary, the problem with experts is that they are so expensive to train and so rare. It is easier to collect data, train the AI and then equip doctors all over the world with it than to have thousands of experts in that particular field.

A doctor that treats patients all day long doesn't have time to keep up with the research and state of the art. A researcher that is on the cutting edge of medicine doesn't have time to treat the patients. We need to equip doctors with AIs to keep them up to date with the best practices.

Re: The Limitations of Deep Learning

#104

Post by sadlion »

kmicklas wrote: post #50
cm2187 wrote: post #42

I think the requirement for a large amount of data is the biggest objection to the reflex "AI will replace [insert your profession here] soon" that many techies, in particular on HN, have. There are many professions where there is very little data available to learn from. In some case (self-driving), companies will invest large amount of money to build this data, by running lots of test self-driving cars, or paying p…

> and will keep doing so until the cost of developing and maintaining software or AI has dropped to zero. I have no idea about the progress of AI, but normal software will get an order of magnitude cheaper to develop as we slowly wake up from the Unix/worse-is-better/everything-is-text mindset and abandon the dynamically typed and imperative languages, broken systems abstractions, etc. that hold us back.

I sincerely would like to know what you think the alternatives are?

Re: The Limitations of Deep Learning

#105

Post by visarga »

pc2g4d wrote: post #28

Programmers contemplating the automation of programming: "To lift some of these limitations and start competing with human brains, we need to move away from straightforward input-to-output mappings, and on to reasoning and abstraction. A likely appropriate substrate for abstract modeling of various situations and concepts is that of computer programs. We have said before (Note: in Deep Learning with Python) that mach…

The author said in a Twitter conversation today that he is aware that this phrase is ignoring something essential - namely, that we have systems with memory and attention. That is something different than simple X to y mappings. With memory you can do general computation, recursivity, graphs, anything. They work well on some problems such as translation, but still need to become much better in order to match general purpose programming. But at least we're past the X->y phase.

Re: The Limitations of Deep Learning

#106

Post by kowdermeister »

> In short, deep learning models do not have any understanding of their input, at least not in any human sense. Our own understanding of images, sounds, and language, is grounded in our sensorimotor experience as humans—as embodied earthly creatures.

Well maybe we should train systems with all our sensory inputs first, like newborns leans about the world. Then make these models available open source like we release operating systems so others can build on top of that.

For example we have ImageNet, but we don't have WalkNet, TasteNet, TouchNet, SmellNet, HearNet... or other extremely detailed sensory data recorded for an extended time. And these should be connected to match the experiences. At least I have no idea they are out there :)

Re: The Limitations of Deep Learning

#108

Post by s-macke »

mojomark wrote: post #99
s-macke wrote: post #72

Earlier quoted context omitted.

You might find a slide of my talk interesting: https://ibb.co/fXAn4a You have to read it from left to right with an twinking eye of course ;)

In your slide - why is back propogation a further stretch from a true bio-NN than an ANN without back propogation?

An ANN still resembles major features of an bio-NN.

1. A network

2. Flow of information is mainly unidirectional through a node

3. Multiple inputs, but one output, which is connected to the inputs of other neurons.

4. The connection strength between 2 neurons can be changed.

5. Non-linear behavior.

After all, I think, this is not such a bad first approximation. Hence the picture in the middle.

But I cannot believe that we learn by comparing thousands or millions of input and output patterns and back propagate the error through the network to perform a gradient descent at the neurons. That is simply not, what our brain does.

Re: The Limitations of Deep Learning

#109

Post by nilkn »

visarga wrote: post #96
language wrote: post #53

Earlier quoted context omitted.

"Machine learning models have no access to such experiences and thus cannot "understand" their inputs in any human-relatable way" It may be that distinctions like the one you're describing here are useful to make, but I don't think this claim refutes the possibility of ML "fitting a particular piece within a larger, yet unarticulated model." I think the assertion is more that our current ways of representing elements…

The problem with neural nets is that they have a fixed input type - tensors or sequences. For example, imagine the task is to count objects in an image and say if the number of red objects is equal to the number of green objects. You make a net that solves this situation. Then you want to change the colors, or add an extra color, and it will fail. Why - because it learns a fixed input representation. What neural nets…

There is actually work being done on this problem, at least to some extent. A DNC, for instance, can accept variable-structure inputs by storing each piece in its external memory bank. This is illustrated in the original Nature paper by feeding in an arbitrary graph definition piece by piece, then feeding in a query about the graph.

This doesn't necessarily address all the nuances of your post, but I do believe it's a step in the right direction. It pushes networks from

"learn how to solve this completely statically defined problem via sophisticated pattern matching"

to

"learn how to interpret a query, drawn from some restricted class of possible queries; accept variable-structure input to the query; strategize about techniques for answering the query; and finally compute the answer, possibly over multiple time-steps"

Re: The Limitations of Deep Learning

#110

Post by mojomark »

rhaps0dy wrote: post #44
xamuel wrote: post #14

Earlier quoted context omitted.

Or maybe increasing NN size/complexity is the 21st century version of adding epicycles to make geocentrism work. http://wiki.c2.com/?AddingEpicycles

Heh, but it makes geocentrism works better! And we don't yet know how 21st century heliocentrism will look like, while adding epicycles is less daunting.

Yay, I found the rabbit hole - technically, no cellestial body rotates purely around the other (thanks mass!). So, perhaps adding epicycles wasn't erronious after all - just a measurement from a different reference point.
Post reply on HN