Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Limitations of Deep Learning

blog.keras.io

Post reply on HN
Print view View original

91–100 of 296 posts

Re: The Limitations of Deep Learning

#91

Post by visarga »

kxyvr wrote: post #71
pishpash wrote: post #20

Earlier quoted context omitted.

Maybe everything is "curve fitting." -- Note: I think it's more hierarchical than that but curve fitting is certainly one of the important capabilities of biological systems.

I don't think so. There's an incredibly important art and science to model selection that is not encapsulated in curve fitting. For example, say we observe a boy throwing a ball and we want to predict where the ball will land. From basic physics, we know the model is `y = 0.5 a t^2 + v0 t + y0` where `a` is the acceleration due to gravity, `v0` is the initial velocity, and `y0` is the initial height. After observing…

> Eventually, it will work, but how much more data do we need?

For a model that small, with so little variance (assume you measure correctly where the ball lands) it would be enough to do just a few throws to fit the parameters.

Re: The Limitations of Deep Learning

#92

Post by yters »

paganel wrote: post #58
kensoh wrote: post #30

Earlier quoted context omitted.

I don't have ML or deep learning background (no Masters or PhD), adding comment from experience with backtesting trading systems. We will collect market data and design algorithms that seem to produce the kind of outcomes we want. Then test on some other data sets which the algorithms have never been applied on. Many iterations later, you can get a decent profitable algorithm. And if the 'holy grail' algo is run in m…

> The quality of the algo and I assume the deep learning model lies in the quality (breadth and depth) of the data, and how honest with himself the person choose to model it. I've only dabbled with machine-learning here and there for the past 10 years or so, but if there's one thing I've learned so far is that the data behind your ML code (and the way it is structured) is responsible for almost all the success or fai…

The data processing inequality says processing data does not increase its information content.

Re: The Limitations of Deep Learning

#93

Post by emodendroket »

kmicklas wrote: post #50
cm2187 wrote: post #42

I think the requirement for a large amount of data is the biggest objection to the reflex "AI will replace [insert your profession here] soon" that many techies, in particular on HN, have. There are many professions where there is very little data available to learn from. In some case (self-driving), companies will invest large amount of money to build this data, by running lots of test self-driving cars, or paying p…

> and will keep doing so until the cost of developing and maintaining software or AI has dropped to zero. I have no idea about the progress of AI, but normal software will get an order of magnitude cheaper to develop as we slowly wake up from the Unix/worse-is-better/everything-is-text mindset and abandon the dynamically typed and imperative languages, broken systems abstractions, etc. that hold us back.

I'm not holding my breath for that development

Re: The Limitations of Deep Learning

#94

Post by philipkglass »

robotresearcher wrote: post #84
philipkglass wrote: post #41

Earlier quoted context omitted.

Implementing a basic perceptron classifier is an undergrad homework assignment. Biological modeling of neurons is a work of decades: http://www.genesis-sim.org/ https://www.neuron.yale.edu/neuron/what_is_neuron

McCulloch's argument was that perhaps the gross behaviour of a NN as layers of simple transfer functions is where the real action is, and the rest of the details are just gravy. The fact we now give this to undergrads as homework suggests that there was some value to this idea.

Students in computer science may implement a perceptron as a homework problem. Students in biology don't do that, nor do they use perceptrons to learn about brains, because perceptrons bear only faint resemblance to biological neurons. Reproducing important biological features of real neurons requires much more complicated software.

I'm not denigrating perceptrons or other neuro-inspired approaches to classification. I'm just pointing out that perceptrons are not a faithful model of neurons.

Re: The Limitations of Deep Learning

#95

Post by emodendroket »

andreyk wrote: post #37

"Here's what you should remember: the only real success of deep learning so far has been the ability to map space X to space Y using a continuous geometric transform, given large amounts of human-annotated data." This statement has a few problems - there is no real reason to interpret the transforms as geometric (they are fundamentally just processing a bunch of numbers into other numbers, in what sense is this geome…

Presumably that's why the word "just" is in scare quotes.

Re: The Limitations of Deep Learning

#96

Post by visarga »

language wrote: post #53
CountSessine wrote: post #45

Surely we shouldn't rush to anthropomorphize neural networks, but we'd ignoring the obvious if we didn't at least note that neural networks do seem to share some structural similarities with our own brains, at least at a very low level, and that they seem to do well with a lot of pattern-recognition problems that we've traditionally considered to be co-incident with brains rather than logical systems. The article not…

"Machine learning models have no access to such experiences and thus cannot "understand" their inputs in any human-relatable way" It may be that distinctions like the one you're describing here are useful to make, but I don't think this claim refutes the possibility of ML "fitting a particular piece within a larger, yet unarticulated model." I think the assertion is more that our current ways of representing elements…

The problem with neural nets is that they have a fixed input type - tensors or sequences. For example, imagine the task is to count objects in an image and say if the number of red objects is equal to the number of green objects. You make a net that solves this situation. Then you want to change the colors, or add an extra color, and it will fail. Why - because it learns a fixed input representation.

What neural nets need is to change their data format from plain tensors to object-relation graphs. The input of the network is represented as a set of objects that have relations among them, and the network has to be permute invariant to the order of presentation. An implementation is Graph Convolutional Nets. They learn to compose concepts in new ways and once they learned to count, compare, select by color, they can solve any combination of those concepts as well. That way the nets generalize better and transfer knowledge from a problem to the next.

Graphs are able to reduce the complexity of learning a neural net that can perform flexible tasks. But in order to get to even better results, it is necessary to add simulation to the mix. By equipping neural nets with simulators, we can simplify the learning problem (because the net doesn't have to learn the dynamics of the environment as well, just the task at hand). Examples of simulators used in DL are AlphaGo, the Reinforcement Learning applications on Atari Games, protein/drug property prediction, generative adversarial networks (in a way).

The interesting thing is that graphs are natural for simulation. They can represent objects as vertices and relations as edges, and by signal propagation the graph works like a circuit, a simulator, producing the answer. My bet is on graphs + simulators. That's how we get to the next level (abstraction and reasoning). DeepMind seems to be particularly focused on RL, games and recently, relation networks. There is also work on making dynamic routing in neural nets, in fact applying graphs implicitly inside the net, by multiple attention heads.

Re: The Limitations of Deep Learning

#97

Post by emodendroket »

zfrenchee wrote: post #70

My qualm with this article is disappointingly poorly backed up. The author makes claims, but does not justify those claims well enough to convince anyone but people who already agree with him. In that sense, this piece is an opinion piece, masquerading as a science. > This is because a deep learning model is "just" a chain of simple, continuous geometric transformations mapping one vector space into another. All it c…

It seems unfair to level the criticism of being incomplete and not fully explaining all the points given that the lead-in to the piece says it's a book excerpt and doesn't explain a lot of stuff that a reader of the book would already have encountered.

Re: The Limitations of Deep Learning

#98

Post by emodendroket »

deepnotderp wrote: post #55

I'd like to offer a somewhat contrasting viewpoint (although this might not sit well with people): deep nets aren't AGI, but they're pretty damn good. There's mounting evidence that they learn similar to how we do, at least in vision; https://arxiv.org/abs/1706.08606 and https://www.nature.com/articles/srep27755 There's quite a few others but these were the most readily available papers. Are deep nets AGI? No, but th…

Similar, perhaps, but certainly not the same, since no human being, at whatever stage of development, would be confused by the adversarial image of a panda given as an example.

Re: The Limitations of Deep Learning

#99

Post by mojomark »

s-macke wrote: post #72
therajiv wrote: post #5

As someone primarily interested in interpretation of deep models, I strongly resonate with this warning against anthropomorphization of neural networks. Deep learning isn't special; deep models tend to be more accurate than other methods, but fundamentally they aren't much closer to working like the human brain than e.g. gradient boosting models. I think a lot of the issue stems from layman explanations of neural net…

You might find a slide of my talk interesting: https://ibb.co/fXAn4a You have to read it from left to right with an twinking eye of course ;)

In your slide - why is back propogation a further stretch from a true bio-NN than an ANN without back propogation?

Re: The Limitations of Deep Learning

#100

Post by eli_gottlieb »

pishpash wrote: post #76
kxyvr wrote: post #71

Earlier quoted context omitted.

I don't think so. There's an incredibly important art and science to model selection that is not encapsulated in curve fitting. For example, say we observe a boy throwing a ball and we want to predict where the ball will land. From basic physics, we know the model is `y = 0.5 a t^2 + v0 t + y0` where `a` is the acceleration due to gravity, `v0` is the initial velocity, and `y0` is the initial height. After observing…

You're talking about the complexity of the model. If you take a purely input-output view of the world (which by the way, even classical Physics does), every problem _is_ curve fitting in a sufficiently high dimensional space. There is no _conceptual_ problem here. There is perhaps a complexity problem, but that's why I wrote that "I think it's more hierarchical than that."

>If you take a purely input-output view of the world (which by the way, even classical Physics does), every problem _is_ curve fitting in a sufficiently high dimensional space.

Not all spaces are Euclidean, and "purely input-output" still contains a lot of room for counterfactuals that ML models fail to capture.

Post reply on HN