Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Limitations of Deep Learning

blog.keras.io

Post reply on HN
Print view View original

71–80 of 296 posts

Re: The Limitations of Deep Learning

#71

Post by kxyvr »

pishpash wrote: post #20
nerdponx wrote: post #6

Earlier quoted context omitted.

Well said. It's just curve fitting.

Maybe everything is "curve fitting." -- Note: I think it's more hierarchical than that but curve fitting is certainly one of the important capabilities of biological systems.

I don't think so. There's an incredibly important art and science to model selection that is not encapsulated in curve fitting. For example, say we observe a boy throwing a ball and we want to predict where the ball will land. From basic physics, we know the model is `y = 0.5 a t^2 + v0 t + y0` where `a` is the acceleration due to gravity, `v0` is the initial velocity, and `y0` is the initial height. After observing one or two thrown balls, even with error, we can estimate the parameters `a`, `v0`, and `y0` relatively well. Alternatively, we could apply a generic machine learning model to this problem. Eventually, it will work, but how much more data do we need? How many additional parameters do we need? Do the machine learning parameters have physical meaning like those in the original model? In this case, I contend the original model is superior.

Now, certainly, there are cases where we don't have a good or known model and machine learning is an extremely important tool for analyzing these cases. However, the process of making this determination and choosing what model to use is not solved by curve fitting or machine learning. This is a decision made by a person. Perhaps some day that will change, and that will be a major advance in intelligent systems, but we don't have that now and it's not clear to me how extending existing methods will lead us there.

Basically, I agree with the sentiment of the grandparent post. Machine learning is largely just curve fitting. How and when to apply a machine learning model vs another model is currently a decision left up to the user.

Re: The Limitations of Deep Learning

#72

Post by s-macke »

therajiv wrote: post #5

As someone primarily interested in interpretation of deep models, I strongly resonate with this warning against anthropomorphization of neural networks. Deep learning isn't special; deep models tend to be more accurate than other methods, but fundamentally they aren't much closer to working like the human brain than e.g. gradient boosting models. I think a lot of the issue stems from layman explanations of neural net…

You might find a slide of my talk interesting:

https://ibb.co/fXAn4a

You have to read it from left to right with an twinking eye of course ;)

Re: The Limitations of Deep Learning

#73

Post by CountSessine »

eli_gottlieb wrote: post #63
CountSessine wrote: post #45

Surely we shouldn't rush to anthropomorphize neural networks, but we'd ignoring the obvious if we didn't at least note that neural networks do seem to share some structural similarities with our own brains, at least at a very low level, and that they seem to do well with a lot of pattern-recognition problems that we've traditionally considered to be co-incident with brains rather than logical systems. The article not…

>In particular, I'm thinking of Dual Process Theory Which has been at least partly debunked as psychology's replication crisis went on, and has been called into question on the neuroscientific angle as well.

Partly, yes - especially with ego depletion on the ropes. I'm not sure that dual process theory needs to be thrown out along with ego depletion, though.

Re: The Limitations of Deep Learning

#74

Post by [deleted] »

eli_gottlieb wrote: post #63
CountSessine wrote: post #45

Surely we shouldn't rush to anthropomorphize neural networks, but we'd ignoring the obvious if we didn't at least note that neural networks do seem to share some structural similarities with our own brains, at least at a very low level, and that they seem to do well with a lot of pattern-recognition problems that we've traditionally considered to be co-incident with brains rather than logical systems. The article not…

>In particular, I'm thinking of Dual Process Theory Which has been at least partly debunked as psychology's replication crisis went on, and has been called into question on the neuroscientific angle as well.

[deleted]

Re: The Limitations of Deep Learning

#76

Post by pishpash »

kxyvr wrote: post #71
pishpash wrote: post #20

Earlier quoted context omitted.

Maybe everything is "curve fitting." -- Note: I think it's more hierarchical than that but curve fitting is certainly one of the important capabilities of biological systems.

I don't think so. There's an incredibly important art and science to model selection that is not encapsulated in curve fitting. For example, say we observe a boy throwing a ball and we want to predict where the ball will land. From basic physics, we know the model is `y = 0.5 a t^2 + v0 t + y0` where `a` is the acceleration due to gravity, `v0` is the initial velocity, and `y0` is the initial height. After observing…

You're talking about the complexity of the model. If you take a purely input-output view of the world (which by the way, even classical Physics does), every problem _is_ curve fitting in a sufficiently high dimensional space. There is no _conceptual_ problem here. There is perhaps a complexity problem, but that's why I wrote that "I think it's more hierarchical than that."

Re: The Limitations of Deep Learning

#77

Post by SomeStupidPoint »

kxyvr wrote: post #71
pishpash wrote: post #20

Earlier quoted context omitted.

Maybe everything is "curve fitting." -- Note: I think it's more hierarchical than that but curve fitting is certainly one of the important capabilities of biological systems.

I don't think so. There's an incredibly important art and science to model selection that is not encapsulated in curve fitting. For example, say we observe a boy throwing a ball and we want to predict where the ball will land. From basic physics, we know the model is `y = 0.5 a t^2 + v0 t + y0` where `a` is the acceleration due to gravity, `v0` is the initial velocity, and `y0` is the initial height. After observing…

You seem to have replied on a tangent: how is what you describe not just "curve fitting"?

Humans didn't magic that model up: you're ignoring the huge amount of human effort over thousands of years that it took to arrive at that model. If we gave a ML algorithm a similar amount of time and asked it to construct a simple model of the situation, it might very well hand back the formula you presented.

Your entire post basically begs the question: it supposes that humans are doing something that isn't "curve fitting", and then uses that to argue that they do more.

What, specifically, are you supposing can't be done by "curve fitting"?

Re: The Limitations of Deep Learning

#78

Post by bitL »

Klockan wrote: post #43
curiousgal wrote: post #33

Earlier quoted context omitted.

To be fair, we do understand how neurons work, at least on a singular level. Perceptrons model that quite well.

But how does a neuron decide to grow new axons or how to change input weights? Biological neurons does this when solving tasks and not just during training. Isn't it possible that human-like intelligence depends on the network being dynamic? For example, when you play a game for the first time a lot of things suddenly starts to click, couldn't that be the result of new connections forming or at least some weights bei…

Biological neurons are fundamentally different to models used in deep learning. They can have multiple outputs, can span over whole brain and do local protein-based computations we don't really understand yet. What we have in perceptron is just a very simple model based on what we observed using rudimentary electricity detectors.

Re: The Limitations of Deep Learning

#79

Post by denzil_correa »

paganel wrote: post #58
kensoh wrote: post #30

Earlier quoted context omitted.

I don't have ML or deep learning background (no Masters or PhD), adding comment from experience with backtesting trading systems. We will collect market data and design algorithms that seem to produce the kind of outcomes we want. Then test on some other data sets which the algorithms have never been applied on. Many iterations later, you can get a decent profitable algorithm. And if the 'holy grail' algo is run in m…

> The quality of the algo and I assume the deep learning model lies in the quality (breadth and depth) of the data, and how honest with himself the person choose to model it. I've only dabbled with machine-learning here and there for the past 10 years or so, but if there's one thing I've learned so far is that the data behind your ML code (and the way it is structured) is responsible for almost all the success or fai…

> I've learned so far is that the data behind your ML code (and the way it is structured) is responsible for almost all the success or failure of any given ML algorithm

Data is indeed a necessary condition but certainly not sufficient. You require a good marriage between engineering features and data to have a good success rate. Learning curves [0] are a good way to understand if your ML algorithm requires more data or better feature engineering.

[0] http://mlwiki.org/index.php/Learning_Curves

Re: The Limitations of Deep Learning

#80

Post by _xnmw »

beachbum8029 wrote: post #56

Pretty interesting that he says reasoning and long term planning are impossible tasks for a neural net, when those tasks are done by billions of neural nets every day. :^)

Pretty interesting that you seem to have conclusively determined that our brain is nothing but a neural net.
Post reply on HN