Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Limitations of Deep Learning

blog.keras.io

Post reply on HN
Print view View original

121–130 of 296 posts

Re: The Limitations of Deep Learning

#121

Post by CountSessine »

eli_gottlieb wrote: post #114
CountSessine wrote: post #73

Earlier quoted context omitted.

Partly, yes - especially with ego depletion on the ropes. I'm not sure that dual process theory needs to be thrown out along with ego depletion, though.

I can see three reasons to "throw it out": 1) Replication failure, plain and simple. 2) Overfitting. There are dozens to hundreds of "cognitive biases" on lists : https://en.wikipedia.org/wiki/List_of_cognitive_biases . When you have hundreds of individual points, you really ought to draw some principles, and the principle should not be, "The system generating all this is rigid and inflexible." 3) Imprecision! Again,…

1) Replication failure, plain and simple.

How many meta-analyses have been conducted as of 2017 showing one result or the other? I don't think ego depletion itself has been thoroughly "debunked" yet. If it is a real effect, it's probably quite small - but I don't think that ego depletion has been thrown in the bin just yet.

2) Overfitting. There are dozens to hundreds of "cognitive biases" on lists: https://en.wikipedia.org/wiki/List_of_cognitive_biases. When you have hundreds of individual points, you really ought to draw some principles, and the principle should not be, "The system generating all this is rigid and inflexible."

3) Imprecision! Again, dozens to hundreds of cognitive biases. What possible behavior or cognitive performance can't be assimilated into the heuristics and biases theory? What can falsify it overall, even after so many of its individual supporting experiments and predictions have fallen down?

Wait a second - has anyone ever tried to explain the "IKEA Effect" using Dual Process Theory? What does a laundry-list of supposed cognitive biases have to do with the theory? Is anyone really trying to explain/predict all this almanac-of-cognitive-failings with Dual Process?

Re: The Limitations of Deep Learning

#122

Post by goatlover »

sadlion wrote: post #104
kmicklas wrote: post #50

Earlier quoted context omitted.

> and will keep doing so until the cost of developing and maintaining software or AI has dropped to zero. I have no idea about the progress of AI, but normal software will get an order of magnitude cheaper to develop as we slowly wake up from the Unix/worse-is-better/everything-is-text mindset and abandon the dynamically typed and imperative languages, broken systems abstractions, etc. that hold us back.

I sincerely would like to know what you think the alternatives are?

Sounds something like Haskell with a Smalltalk environment. Functional, statically typed with powerful type extensions, but with an image instead of text files that you modify.

From just using Jupyter Notebooks, I can see the appeal of working with a live environment, and it's just a fancy REPL, not a full Lisp or Smalltalk environment.

Re: The Limitations of Deep Learning

#123

Post by meh2frdf »

graycat wrote: post #66

On the limitations of machine learning as in the OP, the OP is correct. So, right, current approaches to "machine learning* as in the OP have some serious "limitations". But this point is a small, tiny special case of something else much larger and more important: Current approaches to "machine learning" as in the OP are essentially some applied math, and applied math is commonly much more powerful than machine learn…

Wat?

Re: The Limitations of Deep Learning

#124

Post by [deleted] »

graycat wrote: post #66

On the limitations of machine learning as in the OP, the OP is correct. So, right, current approaches to "machine learning* as in the OP have some serious "limitations". But this point is a small, tiny special case of something else much larger and more important: Current approaches to "machine learning" as in the OP are essentially some applied math, and applied math is commonly much more powerful than machine learn…

[deleted]

Re: The Limitations of Deep Learning

#125

Post by eli_gottlieb »

CountSessine wrote: post #121
eli_gottlieb wrote: post #114

Earlier quoted context omitted.

I can see three reasons to "throw it out": 1) Replication failure, plain and simple. 2) Overfitting. There are dozens to hundreds of "cognitive biases" on lists : https://en.wikipedia.org/wiki/List_of_cognitive_biases . When you have hundreds of individual points, you really ought to draw some principles, and the principle should not be, "The system generating all this is rigid and inflexible." 3) Imprecision! Again,…

1) Replication failure, plain and simple. How many meta-analyses have been conducted as of 2017 showing one result or the other? I don't think ego depletion itself has been thoroughly "debunked" yet. If it is a real effect, it's probably quite small - but I don't think that ego depletion has been thrown in the bin just yet. 2) Overfitting. There are dozens to hundreds of "cognitive biases" on lists: https://en.wikipe…

>Is anyone really trying to explain/predict all this almanac-of-cognitive-failings with Dual Process?

To my understanding, yes. That's basically what Dual Process theories exist for: to separate the brain into heuristic/bias processing as one process, and computationally expensive model-based cause-and-effect reasoning as another process. Various known cognitive processes or results are then sort of classified on one side of the line or another.

When you apply Dual Process paradigms to specific corners of cognition, they can be useful. For example, I've seen papers purporting to show that measured uncertainty allows model-free and model-based reinforcement learning algorithms to trade off decision-making "authority". This is less elegant than an explicitly precision-measuring free-energy counterpart, but it's still a viable hypothesis about how the brain can implement a form of bounded rationality when bounded in both sample data and compute power.

But when you scale Dual Processes up to a whole-brain theory, it's just too good at describing anything that involves dichotomizing into a "fast-and-frugal" form of processing and another expensive, reconstructive form of processing. One of the big issues here is that besides the potentially false original evidence for Dual Processes, we don't necessarily have reason to believe there exists any dichotomy, rather than a more continuous tradeoff between frugal heuristic processing and difficult reconstructive processing. The precision-weighting model-selection theory actually makes much more sense here.

Re: The Limitations of Deep Learning

#126

Post by raverbashing »

romaniv wrote: post #87
therajiv wrote: post #5

As someone primarily interested in interpretation of deep models, I strongly resonate with this warning against anthropomorphization of neural networks. Deep learning isn't special; deep models tend to be more accurate than other methods, but fundamentally they aren't much closer to working like the human brain than e.g. gradient boosting models. I think a lot of the issue stems from layman explanations of neural net…

I think it would help a lot if we brought random forests and SVMs to the same level of performance as DNNs. Demonstrating that more "mechanical" algorithms can be as efficient would dispel some of the anthropomorphism and allow for better analysis of why certain things work. I also believe that researches have responsibility to outline the limits of their own algorithms in research papers. (For example, presenting ex…

Not possible, unfortunately

Re: The Limitations of Deep Learning

#127

Post by cm2187 »

goatlover wrote: post #122
sadlion wrote: post #104

Earlier quoted context omitted.

I sincerely would like to know what you think the alternatives are?

Sounds something like Haskell with a Smalltalk environment. Functional, statically typed with powerful type extensions, but with an image instead of text files that you modify. From just using Jupyter Notebooks, I can see the appeal of working with a live environment, and it's just a fancy REPL, not a full Lisp or Smalltalk environment.

If it has to be a general public language, I'm afraid it will have to be light on special characters and abbreviations or acronyms that made sense 30 years ago. I'd say a Basic or Python-like language, but modernised, and with strong typing to enable the IDE to help a lot the users with auto-completion and error checking.

But if you think about it, most business users are even intimidated by VBA. So it will have to be very fluffy, and I don't think you can spare the mandatory coding 101 teaching at school.

Re: The Limitations of Deep Learning

#128

Post by meh2frdf »

Correct me if I'm wrong but I don't see that with 'deep learning' we have answered/solved any of the philosophical problems of AI that existed 25 years ago (stopped paying attention about then).

Yes we have engineered better NN implementations and have more compute power, and thus can solve a broader set of engineering problems with this tool, but is that it?

Re: The Limitations of Deep Learning

#129

Post by sqeaky »

s-macke wrote: post #108
mojomark wrote: post #99

Earlier quoted context omitted.

In your slide - why is back propogation a further stretch from a true bio-NN than an ANN without back propogation?

An ANN still resembles major features of an bio-NN. 1. A network 2. Flow of information is mainly unidirectional through a node 3. Multiple inputs, but one output, which is connected to the inputs of other neurons. 4. The connection strength between 2 neurons can be changed. 5. Non-linear behavior. After all, I think, this is not such a bad first approximation. Hence the picture in the middle. But I cannot believe th…

When there is feedback in neurons, what do you think that conveys?

I agree it is not some simple error correction like what is propagated backwards, but it happens often and I presume its something useful or it wouldn't be there.

Re: The Limitations of Deep Learning

#130

Post by CountSessine »

eli_gottlieb wrote: post #125
CountSessine wrote: post #121

Earlier quoted context omitted.

1) Replication failure, plain and simple. How many meta-analyses have been conducted as of 2017 showing one result or the other? I don't think ego depletion itself has been thoroughly "debunked" yet. If it is a real effect, it's probably quite small - but I don't think that ego depletion has been thrown in the bin just yet. 2) Overfitting. There are dozens to hundreds of "cognitive biases" on lists: https://en.wikipe…

>Is anyone really trying to explain/predict all this almanac-of-cognitive-failings with Dual Process? To my understanding, yes. That's basically what Dual Process theories exist for: to separate the brain into heuristic/bias processing as one process, and computationally expensive model-based cause-and-effect reasoning as another process. Various known cognitive processes or results are then sort of classified on one…

This is a fantastic answer - thank you, Eli. So what do you think of the original article?
Post reply on HN