Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Limitations of Deep Learning

blog.keras.io

Post reply on HN
Print view View original

251–260 of 296 posts

Re: The Limitations of Deep Learning

#251

Post by red75prime »

landon32 wrote: post #234
red75prime wrote: post #204

Earlier quoted context omitted.

> but it looks that my data-related advice falls on deaf ears every time, he's only interested in me pointing to him the latest cool ML algorithm. So, it seems their learning/planning algorithm fails, even when it is given the right data. That's unfortunate. Sorry, I can't help but notice that you aren't happy with their brain's algorithm, while talking about importance of data. I don't say that data doesn't matter o…

Could actually be their data, right? Imagine if you had only had experience with software engineering. The only data you use when engineering software are the data you learn when using the product or writing tests, it's all the algorithms behind it that's important. So to them, they just don't have data on situations where the data are important. Wow that's confusing wording. I hope it makes sense.

It does, but the algorithm doesn't seems to be state-of-the-art, it's more like current ML algorithms, which need lots of data to work successfully in each new domain. Well, there's a lot of improvement possibilities, at least.

Re: The Limitations of Deep Learning

#252

Post by psandersen »

raverbashing wrote: post #250
fnl wrote: post #185

Earlier quoted context omitted.

True - until some clever guy proves us all wrong and finds ways to train some multidimensional/complex/deep/... kernel/forest/swarm/... that can learn those nonlinearities that currently only deep nets can be trained to detect (essentially, due to their relative simplicity, I'd say) :-)

I don't think we'll see a deep svm, but if we see one I think we'll have something very powerful Same for a deep decision tree (forest?). Or maybe a combination of several techniques, etc

Probably comes down to whether the model can be trained with gradient descent (at least in the short term).

A general pre-trained RL guided architecture search (#1) together with more choices of nonlinearity (#2), feature extraction (#3), pooling and memory argumentation (#4) and other tricks (#5) could be very powerful amongst many domains. Make it be able to accept multiple pre-trained models as priors and we're well on our way to general AI or at least a place where most data-scientists could be automated away.

(#1 deepmind had a demo a year back or so that was quite novel) (#2 vaguely remember someone training decision trees with gradient descent; could definitely see a 'randomforest' layer appearing in the middle of deep nets) (#3 just convolutions + tricks really). (#4 neural turing machine etc) (#5 any attention mechanism/any sequence mechanism (rnn/lstm etc)/ any graph relational understanding like the recent deepmind paper).

Re: The Limitations of Deep Learning

#253

Post by ThomPete »

posterboy wrote: post #249
ThomPete wrote: post #235

Earlier quoted context omitted.

About a decade ago winning GO or self-driving cars were seen as pipedreams many decades away. Yet here we are. The author is making the mistake of thinking that just because he can show some areas were we aren't as far as we thought he has made an argument against AI. Thats not how it works. We don't get to decide what is the right metrics. All we can see is that we keep making progress sometimes large leaps sometime…

The fallacy there is glorifying consciousness. Full consciousness as in omniscence is an unachievable ideal. If we prescribe consciousness to ourselves, depending on the individual theory of conscious thought, that's likely faulty in some respect already.

I don't see anyone glorifying consciousness especially not as some omniscient ideal. In fact I only see people arguing that consciousness isn't really the goal or the focus here but rather that you can't talk with any certainty about whether or not it's possible. You can however point to the fact that we are making progress towards more and more complex relationships and that this looks very much like how we became conscious. Thats all really.

Re: The Limitations of Deep Learning

#254

Post by nerdponx »

kxyvr wrote: post #90
SomeStupidPoint wrote: post #77

Earlier quoted context omitted.

You seem to have replied on a tangent: how is what you describe not just "curve fitting"? Humans didn't magic that model up: you're ignoring the huge amount of human effort over thousands of years that it took to arrive at that model. If we gave a ML algorithm a similar amount of time and asked it to construct a simple model of the situation, it might very well hand back the formula you presented. Your entire post ba…

I believe the process for deriving fundamental physical models differs from the techniques used in ML. For example, say we want to use the principle of least action to derive an expression for energy similar to what Landau and Lifshitz derive in their book Mechanics. Here, we assume that the motion of a particle is defined by its position and velocity. We assume that the motion of the particle is defined by an optimi…

Of course. "What do I fit this curve to" is a prerequisite to "what is the shape of this curve?"

You shouldn't feel the need to defend theory-based modeling against some imagined incursion from arrogant deep learning researchers. NNs work tremendously well in a few specific problem domains that we had no way to approach otherwise. Elsewhere, they're not much better than any other prediction algorithm. By the way XGBoost is curve-fitting, too.

Re: The Limitations of Deep Learning

#255

Post by nerdponx »

kxyvr wrote: post #131
SomeStupidPoint wrote: post #117

Earlier quoted context omitted.

You haven't explained how the first case isn't "curve fitting": the agents performing the compilation of those facts into the new fact are just spitting out the "best" fit string of symbols based on learned rules, etc etc. Somethings computers can (theoretically) do, and which fits the description "curve fitting" just fine. School (and other education) is training the model they're using to do that compilation, but i…

Alright, so from my perspective, curve fitting consists of three things 1. Definition of a model. ML models like multilayer perceptrons used a superposition of sigmoids, but newer models have superpositions of other functions and more nested hierarchies. 2. A metric to define misfit. Most of the time we use least squares because it's differentiable, but other metrics are possible. 3. An optimization algorithm to mini…

I think you're reading way too far into my post. I was just pointing out that our amazing AI revolution is really just a new type of function approximation being that has magical-seeming results.

Re: The Limitations of Deep Learning

#256

Post by pron »

> Doing this well is a game-changer for essentially every industry, but it is still a very long way from human-level AI.

We're still a long way from even insect level "intelligence" (if it could even be called that), hence the harm in calling it AI in the first place. The fact that machine learning performs some particular tasks better than humans means little. That was true of computers since their inception. The question of how much closer we are to human-level AI than to the starting point of machine learning and neural networks over 70 years ago is very much an open question. That after 70 years of research into neural networks in particular and to machine learning in general, we are still far from insect-level intelligence makes anyone suggesting a timeline for human-level AI sound foolish (although hypothetically, the leap from insect-level intelligence to human-level could be technically simple, but we really have no idea).

Re: The Limitations of Deep Learning

#257

Post by skgoa »

honestoHeminway wrote: post #244
denzil_correa wrote: post #79

Earlier quoted context omitted.

> I've learned so far is that the data behind your ML code (and the way it is structured) is responsible for almost all the success or failure of any given ML algorithm Data is indeed a necessary condition but certainly not sufficient. You require a good marriage between engineering features and data to have a good success rate. Learning curves [0] are a good way to understand if your ML algorithm requires more data…

Much of the programming with ML has moved towards cleaning, extrapolating and generating the data. But this type of programing is - miracles- bugfree. We never hear of data-conversion gone wrong, data corrupted or data-mining withou conclusive results here. Obviously such bugs lack the glamour of security bugs.

It's also very difficult to catch these errors. Your trained model just doesn't work as well as it could, but how would you be able to tell?

Re: The Limitations of Deep Learning

#258

Post by skgoa »

Florin_Andrei wrote: post #54
therajiv wrote: post #5

As someone primarily interested in interpretation of deep models, I strongly resonate with this warning against anthropomorphization of neural networks. Deep learning isn't special; deep models tend to be more accurate than other methods, but fundamentally they aren't much closer to working like the human brain than e.g. gradient boosting models. I think a lot of the issue stems from layman explanations of neural net…

> Pretty much every time DL is covered by media, there has to be some contrived comparison to human brains Well, what we've done so far is emulate maybe 1 mm^3 of brain matter - some isolated, very specialized functional blocks in the greater architecture of the brain. They behave as expected - are experts on very narrow topics, but of course fail to integrate their functioning with a larger body of knowledge, becaus…

You seem to be under the assumption that a typical feedforward DNN is anywhere close to operating like the brain, just on a smaller scale. But that assumption is not correct.

Both the brain and artificial neural networks are connectivist, but that's about where the similarities end. The brain uses completely unknown algorithms and mechanisms that are almost certainly very different from our (current) ANNs. So it's not just a matter of increasing the scale.

Re: The Limitations of Deep Learning

#259

Post by emodendroket »

Buttons840 wrote: post #146
paganel wrote: post #58

Earlier quoted context omitted.

> The quality of the algo and I assume the deep learning model lies in the quality (breadth and depth) of the data, and how honest with himself the person choose to model it. I've only dabbled with machine-learning here and there for the past 10 years or so, but if there's one thing I've learned so far is that the data behind your ML code (and the way it is structured) is responsible for almost all the success or fai…

> focus a lot of his time on the data itself... from where he intends to collect it? how is it structured? is it reliable? is it "enough"? What's the best books on this subject? I suppose it's a very broad topic and thus more difficult to talk about than a single "neural network" algorithm.

This is the domain of statistics, isn't it?

Re: The Limitations of Deep Learning

#260

Post by mehh »

brittohalloran wrote: post #202
landon32 wrote: post #182

Earlier quoted context omitted.

This is totally true, but I think it's still important to note that while something like Artificial General Intelligence is still way beyond the state of the art, the state of the art still has a huge impact on the world. A tiny slice of that can be seen in autonomous vehicles and the impact that they seem poised to have.

Don't underestimate the self fulfilling prophecy effect. Quite possible that the massive influx into the field right now will move the needle.

I fully agree with the above responses, and I am optimistic about major break throughs, however like many of you guys, I don't think we should just assume a bit more horsepower and things will magically work.
Post reply on HN