Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Limitations of Deep Learning

blog.keras.io

Post reply on HN
Print view View original

201–210 of 296 posts

Re: The Limitations of Deep Learning

#201

Post by eli_gottlieb »

nerdponx wrote: post #192
eli_gottlieb wrote: post #100

Earlier quoted context omitted.

>If you take a purely input-output view of the world (which by the way, even classical Physics does), every problem _is_ curve fitting in a sufficiently high dimensional space. Not all spaces are Euclidean, and "purely input-output" still contains a lot of room for counterfactuals that ML models fail to capture.

What do you mean by counterfactuals? NNs are function approximation algorithms, in any geometry. No ifs ands or buts about it.

Oh, I agree that neural networks are function approximators with respect to some geometry. When I say "counterfactuals", I'm talking about typical Bayes-net style counterfactuals, but as also used in cognitive psychology. We know that human minds evaluate counterfactual statements in order to test and infer causal structure. We thus know that neural networks are insufficient for "real" cognition.

Re: The Limitations of Deep Learning

#202

Post by brittohalloran »

landon32 wrote: post #182
nv-vn wrote: post #173

Earlier quoted context omitted.

Yep. The whole machine learning craze is just fueled by the fact that it's now feasible to create models for handwriting/voice/image recognition that actually work reliably. But in terms of the underlying technology, we haven't had some "breakthrough" that explained how the brain works or anything even close to that.

This is totally true, but I think it's still important to note that while something like Artificial General Intelligence is still way beyond the state of the art, the state of the art still has a huge impact on the world. A tiny slice of that can be seen in autonomous vehicles and the impact that they seem poised to have.

Don't underestimate the self fulfilling prophecy effect. Quite possible that the massive influx into the field right now will move the needle.

Re: The Limitations of Deep Learning

#203

Post by tlear »

paganel wrote: post #58
kensoh wrote: post #30

Earlier quoted context omitted.

I don't have ML or deep learning background (no Masters or PhD), adding comment from experience with backtesting trading systems. We will collect market data and design algorithms that seem to produce the kind of outcomes we want. Then test on some other data sets which the algorithms have never been applied on. Many iterations later, you can get a decent profitable algorithm. And if the 'holy grail' algo is run in m…

> The quality of the algo and I assume the deep learning model lies in the quality (breadth and depth) of the data, and how honest with himself the person choose to model it. I've only dabbled with machine-learning here and there for the past 10 years or so, but if there's one thing I've learned so far is that the data behind your ML code (and the way it is structured) is responsible for almost all the success or fai…

This is incredibly true, even with crappy old algorithms you can do A LOT if you have great data.

Recent experience with a company that is building some models based on.. few guys recording few hours of audio and annotating it. I still can't get over the fact that otherwise smart people think this is going to work at all.

Re: The Limitations of Deep Learning

#204

Post by red75prime »

paganel wrote: post #58
kensoh wrote: post #30

Earlier quoted context omitted.

I don't have ML or deep learning background (no Masters or PhD), adding comment from experience with backtesting trading systems. We will collect market data and design algorithms that seem to produce the kind of outcomes we want. Then test on some other data sets which the algorithms have never been applied on. Many iterations later, you can get a decent profitable algorithm. And if the 'holy grail' algo is run in m…

> The quality of the algo and I assume the deep learning model lies in the quality (breadth and depth) of the data, and how honest with himself the person choose to model it. I've only dabbled with machine-learning here and there for the past 10 years or so, but if there's one thing I've learned so far is that the data behind your ML code (and the way it is structured) is responsible for almost all the success or fai…

> but it looks that my data-related advice falls on deaf ears every time, he's only interested in me pointing to him the latest cool ML algorithm.

So, it seems their learning/planning algorithm fails, even when it is given the right data. That's unfortunate.

Sorry, I can't help but notice that you aren't happy with their brain's algorithm, while talking about importance of data. I don't say that data doesn't matter or anything. Just random observation.

Re: The Limitations of Deep Learning

#205

Post by Leilani- »

Cybiote wrote: post #149
mtremsal wrote: post #140

Earlier quoted context omitted.

My understanding is that innovation comes from reinforcement learning during self-play (rather than supervised learning of pro games), and thus goes against the best moves suggested by AlphaGo's policy network, in turn pushing it towards new options. In a sense, it seems innovation arises when the value network forces the policy network to expand the search space because an apparently unlikely move leads to downstrea…

It's not that simple. The creativity is that the combination of rollouts, policy and value networks allow for more efficient traversal of the search space. Which gets you better exploration of possible paths, meaning more options than a human considered and therefore more creativity.

ғιnally, мy paycнecĸ ιѕ $ 8,500? a worĸιng 10 нoυrѕ per weeĸ onlιne. мy вroтнer’ѕ ғrιend нad an average oғ 12ĸ ғor ѕeveral мonтнѕ, нe worĸ aвoυт 22 нoυrѕ a weeĸ. ι can noт вelιeve нow eaѕy ιт ιѕ, once ι тry тo do ѕo. тнιѕ ιѕ wнaт ι do........http://www.fl-y.com/3pbr

Re: The Limitations of Deep Learning

#206

Post by fnl »

logicallee wrote: post #189
fnl wrote: post #183

Earlier quoted context omitted.

That's just another version of the trap GP spoke about. About a decade ago everybody was expecting emergent complex behavior from all kinds of evolutionary, intelligent ("swarm") systems. Didn't happen, seen that. https://en.m.wikipedia.org/wiki/Swarm_intelligence

>Didn't happen, ...yet. See my comment here: https://news.ycombinator.com/item?id=14770230

"Never" is a strong prediction. But yes, ANNs have nothing in common with BNNs (biological ... :-)) at all, other than taking them as a very rough abstraction for teaching the basic intuition of the chained up tensor transformations.

The hard thing is to predict the when, or even if, of AI. If it will happen, it will be a sudden, light-switch like moment. I don't think AI can happen gradually. At least the first artificially scentient entity will be a moment much like a singularity some love to predict in the near future...

But as to when that moment will occur, or even if, I think we have no real data that shows we are any closer today than say 10 or 30 years ago. Pattern matching, no matter how complex, isn't "all there is" to intelligence and conciseness.

EDIT: OP changed his reply from "will never happen" to "hasn't happened yet" while I was replying, explaining why mine might read a bit strange now... :-)

Re: The Limitations of Deep Learning

#207

Post by divideby0829 »

webmaven wrote: post #171
unlikelymordant wrote: post #148

Earlier quoted context omitted.

No they aren't? RNNs have state that gets modified as time goes on. The RNN has to learn what is important to save as state, and how to modify it in response to different inputs. There is no explicit time-stamping.

There is an implicit ordering of timesteps ("before" and "after") though, right? If you have that, you can dispense with an explicit time dimension.

not necessarily, depending on the usage RNN based models are sometimes trained in both directions, i.e. for every sample of say videos show it to the network in its natural time direction and then also reversed. This is motivated some say to eliminate dependence on specific order of sequences but instead to train an integrator.

Re: The Limitations of Deep Learning

#208

Post by divideby0829 »

andreyk wrote: post #37

"Here's what you should remember: the only real success of deep learning so far has been the ability to map space X to space Y using a continuous geometric transform, given large amounts of human-annotated data." This statement has a few problems - there is no real reason to interpret the transforms as geometric (they are fundamentally just processing a bunch of numbers into other numbers, in what sense is this geome…

yeah this was my main problem, I guess he is technically right because they are geometric but many of his analogies like the paper crumpling were deeply misleading as they would imply that the transformations are linear. The fact that they are not is fundamental to neural networks working.

Re: The Limitations of Deep Learning

#209

Post by deepGem »

This is because a deep learning model is "just" a chain of simple, continuous geometric transformations mapping one vector space into another.

Per my understanding - Each vector space represents the full state of that layer. Which is probably why the transformations work for such vector spaces.

A sorting algorithm unfortunately cannot be modeled as a set of vector spaces each representing the full state. For instance, an intermediary state of a quick sort algorithm does not represent the full state. Even if a human was to look at that intermediary step in isolation, they will have no clue as to what that state represents. On the contrary, if you observe the visualized activations of an intermediate layer in VGG , you can understand that the layer represents some elements of an image.

Re: The Limitations of Deep Learning

#210

Post by divideby0829 »

visarga wrote: post #105
pc2g4d wrote: post #28

Programmers contemplating the automation of programming: "To lift some of these limitations and start competing with human brains, we need to move away from straightforward input-to-output mappings, and on to reasoning and abstraction. A likely appropriate substrate for abstract modeling of various situations and concepts is that of computer programs. We have said before (Note: in Deep Learning with Python) that mach…

The author said in a Twitter conversation today that he is aware that this phrase is ignoring something essential - namely, that we have systems with memory and attention. That is something different than simple X to y mappings. With memory you can do general computation, recursivity, graphs, anything. They work well on some problems such as translation, but still need to become much better in order to match general…

considering they're the author of a python based machine learning library I would sure hope so. Still it seems like a pretty grievous oversight in writing the dang thing at all considering how at least in my fields of research memory-ful networks are increasingly popular.
Post reply on HN