Live data from Hacker News

The Limitations of Deep Learning

blog.keras.io

211–220 of 296 posts

Re: The Limitations of Deep Learning

#211

Earlier quoted context omitted.

There is an implicit ordering of timesteps ("before" and "after") though, right? If you have that, you can dispense with an explicit time dimension.

not necessarily, depending on the usage RNN based models are sometimes trained in both directions, i.e. for every sample of say videos show it to the network in its natural time direction and then also reversed. This is motivated some say to eliminate dependence on specific order of sequences but instead to train an integrator.

So, time's arrow can be reversed, and the model can thus extrapolate both forward and backward. Cool!

However, that doesn't actually eliminate the axis/dimension. Eliminating timestamps only makes the dimension a unitless scalar (IOW 'time' tautologically increments at a 'rate' of 'one frame per frame').

Re: The Limitations of Deep Learning

#212

Earlier quoted context omitted.

But it turns out that they don't have to be. We know that radically different low-level implementations can approximate the same higher-level functions given a large enough network and enough training (eg. half-precision floating point, integer, or even binary ANNs, not to mention the wide variety of activation functions such as relu, sigmoid, tanh, maxout, softmax, etc.), and we've seen increasingly varied ANN archi…

I was disagreeing with someone who wrote that we understand how neurons work and that perceptrons model them "quite well." They do not model biological neurons well at all. I agree that biological fidelity is not important for building useful ANNs.

I presented (a vulgar summary of) McCulloch's hypothesis, not my own. And since I didn't use the words "quite well", you are not entitled to put them in quotes.

Re: The Limitations of Deep Learning

#213

Earlier quoted context omitted.

Thanks for the link to this interesting paper. I think we're seeing some recapitulation of those arguments WRT 'ensembles of DL models' approaches.

I agree. Google has come out with some papers that are, to put it harshly, basic gluing together of DL models followed by loads of training on their compute resources.

Not just Google. The FractalNet paper comes to mind.

Re: The Limitations of Deep Learning

#214
This article is a bit misleading. I believe NNs are a lot like the human brain. But just the lowest level of our brain. What psychologists might call "procedural knowledge".

Example: learning to ride a bike. You have no idea how you do it. You can't explain it in words. It requires tons of trial and error. You can give a bike to a physicist that has a perfect deep understanding of the laws of physics. And they won't be any better at riding than a kid.

And after you learn to ride, change the bike. Take one where the handle is inversed. And turning it right turns the wheel left. No matter how good you are at riding a normal bike, no matter how easy it seems it should be, it's very hard. Requires relearning how to ride basically from scratch. And when you are done, you will even have trouble going back to a normal bike. This sounds familiar to the problems of deep reinforcement learning, right?

If you use only the parts of the brain you use to ride a bike, would you be able to do any of the tasks described in the article? E.g. learn to guide spacecraft trajectories with little training, through purely analog controls and muscle memory? Can you even sort a list in your head without the use of pencil and paper?

Similarly recognizing a toothbrush as a baseball bat isn't as bizarre as you think. Most NNs get one pass over an image. Imagine you were flashed that image for just a millisecond. And given no time to process it. No time to even scan it with your eyes! You certain you wouldn't make any mistakes?

But we can augment NNs with attention, with feedback to lower layers from higher layers, and other tricks that might make them more like human vision. It's just very expensive.

And that's another limitation. Our largest networks are incredibly tiny compared to the human brain. It's amazing they can do anything at all. It's unrealistic to expect them to be flawless.

Re: The Limitations of Deep Learning

#215

I'm excited to hear about how we bring about abstraction. I was wondering how a NN would go about discovering F = ma and the laws of motion. As far as I can tell, it has a lot of similarities to how humans would do it. You'd roll balls down slopes like in high school and get a lot of data. And from that you'd find there's a straight line model in there if you do some simple transformations. But how would you come to…

The interesting part of human intelligence isn't the ability to calculate that F=MA based on measurement, it's the ability to come to a social consensus on the meaning of F,M,A, and =, and to decide that the relationship between force, mass, and acceleration would be a useful or interesting thing to know.

Re: The Limitations of Deep Learning

#216

Earlier quoted context omitted.

We can debate, but I don't think another AI winter will happen again in my lifetime. AI work is just earning way too much money for its funding to get cut, and a lot of funding is currently private too.

I wasn't arguing for another AI Winter per-se. My warning was more along the lines of pointing out a potential personal "career winter".

My mothers neighbour is working part time and averaging $9000 a month. I'm a single mum and just got my first paycheck for $6546! I still can't believe it. I tried it out cause I got really desperate and now I couldn't be happier. Heres what I do, •••••••••>> http://cutt.us/7N2sG

Re: The Limitations of Deep Learning

#218
post #87

Earlier quoted context omitted.

I think it would help a lot if we brought random forests and SVMs to the same level of performance as DNNs. Demonstrating that more "mechanical" algorithms can be as efficient would dispel some of the anthropomorphism and allow for better analysis of why certain things work. I also believe that researches have responsibility to outline the limits of their own algorithms in research papers. (For example, presenting ex…

Not possible, unfortunately

Kaggle has already proven hundreds of times over that deep learning is not a silver bullet.

Re: The Limitations of Deep Learning

#219

Earlier quoted context omitted.

> Pretty much every time DL is covered by media, there has to be some contrived comparison to human brains Well, what we've done so far is emulate maybe 1 mm^3 of brain matter - some isolated, very specialized functional blocks in the greater architecture of the brain. They behave as expected - are experts on very narrow topics, but of course fail to integrate their functioning with a larger body of knowledge, becaus…

> Well, what we've done so far is emulate maybe 1 mm^3 of brain matter - some isolated, very specialized functional blocks in the greater architecture of the brain. They behave as expected - are experts on very narrow topics, but of course fail to integrate their functioning with a larger body of knowledge, because that body just isn't there (yet). I think you're falling into the same anthropomorphism trap that the G…

Hey guys i would suggest you to avoid scams and things that are promising you online jobs. If you really want to start making money online, go to ►►►►http://mcaf.ee/4ngo1k . This is a step by step system that will teach you how to earn over 3k per week. This is what worked for me. It only cost a dollar.

Re: The Limitations of Deep Learning

#220
post #5

As someone primarily interested in interpretation of deep models, I strongly resonate with this warning against anthropomorphization of neural networks. Deep learning isn't special; deep models tend to be more accurate than other methods, but fundamentally they aren't much closer to working like the human brain than e.g. gradient boosting models. I think a lot of the issue stems from layman explanations of neural net…

> The model's intuition doesn't work like a human's

The model doesn't have intuition, it is just a series of computations.

Post reply on HN