Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There Will Be a Scientific Theory of Deep Learning

arxiv.org

Post reply on HN
Print view View original

131–140 of 179 posts

Re: There Will Be a Scientific Theory of Deep Learning

#131

Post by psyklic »

hodgehog11 wrote: post #39

As someone who works in the area, this provides a decent summary of the most popular research items. The most useful and impressive part is the set of open problems at the end, which just about covers all of the main research directions in the field. The skepticism I'm seeing in the comments really highlights how little of this work is trickling down to the public, which is very sad to see. While it can offer few mat…

I'm constantly surprised how many people are critical of research to understand neural nets, immediately telling me they are black boxes and hopeless to understand. I believe it's a consequence of being portrayed as the opposite of (classically interpretable) linear regression.

Many people additionally have little patience for research when the engineering is moving so quickly. Even many interpretability researchers give up far too soon if research doesn't yield immediately gratifying results.

Re: There Will Be a Scientific Theory of Deep Learning

#132

Post by Someone »

pash wrote: post #14
RyanShook wrote: post #5

Here's where I'm missing understanding: for decades the idea of neural networks had existed with minimal attention. Then in 2017 Attention Is All You Need gets released and since then there is an exponential explosion in deep learning. I understand that deep learning is accelerated by GPUs but the concept of a transformer could have been used on much slower hardware much earlier.

The inflection point was 2012, when AlexNet [0], a deep convolutional neural net, achieved a step-change improvement in the ImageNet classification competition. After seeing AlexNet’s results, all of the major ML imaging labs switched to deep CNNs, and other approaches almost completely disappeared from SOTA imaging competitions. Over the next few years, deep neural networks took over in other ML domains as well. The…

> Even a mouse needs hundreds of millions of neurons to do what a mouse does.

Under the very light assumption that a mouse doesn’t have neurons it doesn’t need, a mouse needs whatever number of neurons it has to do what a mouse does, so that’s not saying much.

Reading https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n..., an ant has only 250k neurons and many reptiles can do with around 10 million.

That page also says 71 million for the house mouse. So what is it that a mouse does that reptiles do not do that requires them to have that much larger a brain? Caring for their children?

Re: There Will Be a Scientific Theory of Deep Learning

#133

Post by xyhopguy »

quicklywilliam wrote: post #21
RyanShook wrote: post #5

Here's where I'm missing understanding: for decades the idea of neural networks had existed with minimal attention. Then in 2017 Attention Is All You Need gets released and since then there is an exponential explosion in deep learning. I understand that deep learning is accelerated by GPUs but the concept of a transformer could have been used on much slower hardware much earlier.

Agreed, there is probably a theoretical world where we got enough money/compute together and had this explosion happen earlier. Or perhaps a world where it happened later. I think a big part of what enabled the AI boom was the concentration of money and compute around the crypto boom.

not really. early deep learning models were run on single consumer-grade GPUs. the inflection occured _right_ when parallel computing became fast enough to do backprop in a reasonable amount of time with performance better than tree methods.

at that time all the compute resources in the world would not have been enough to train the models from even the last ~6 years or so, probably more.

Re: There Will Be a Scientific Theory of Deep Learning

#134

Post by sdenton4 »

appplication wrote: post #74
hodgehog11 wrote: post #72

Earlier quoted context omitted.

It wouldn't work if your landscape has more local minima than atoms in the known universe (which it does) and only some of them are good. Neural networks can easily fail, but there's a lot of things one can do to help ensure it works.

Not a mathematician so I’m immediately out of my depth here (and butchering terminology), but it seems, intuitively, like the presence of a massive amount of local minima wouldn’t really be relevant for gradient descent. A given local minimum would need to have a “well” at least be as large as your step size to reasonably capture your descent. E.g. you could land perfectly on a local minima but you won’t stay the unl…

The randomness (and exploration) encouraged by batch training also helps avoid 'real' minima, if they exist.

Re: There Will Be a Scientific Theory of Deep Learning

#135

Post by sdenton4 »

Someone wrote: post #132
pash wrote: post #14

Earlier quoted context omitted.

The inflection point was 2012, when AlexNet [0], a deep convolutional neural net, achieved a step-change improvement in the ImageNet classification competition. After seeing AlexNet’s results, all of the major ML imaging labs switched to deep CNNs, and other approaches almost completely disappeared from SOTA imaging competitions. Over the next few years, deep neural networks took over in other ML domains as well. The…

> Even a mouse needs hundreds of millions of neurons to do what a mouse does. Under the very light assumption that a mouse doesn’t have neurons it doesn’t need, a mouse needs whatever number of neurons it has to do what a mouse does, so that’s not saying much. Reading https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n... , an ant has only 250k neurons and many reptiles can do with around 10 million. That pa…

Speed may be a factor - reptiles and mice live their lives at very different paces.

Re: There Will Be a Scientific Theory of Deep Learning

#136

Post by sdenton4 »

imtringued wrote: post #110
jvanderbot wrote: post #27

Earlier quoted context omitted.

If you think a 2 year old is doing deep learning, you're probably wrong. But if you think natural selection was providing end to end loss optimization, you might be closer to right. An _awful lot_ of our brain structure and connectivity is born, vs learned, and that goes for Mice and Men.

I don't know what you mean by end to end loss optimization in particular, but if you mean something that involves global propagation of errors e.g. backpropagation you are dead wrong. Predictive coding is more biologically plausible because it uses local information from neighbouring neurons only.

By end to end loss optimization, they mean evolution: Try a thing, and see if it dies or reproduces more. Repeat until moon landing.

Re: There Will Be a Scientific Theory of Deep Learning

#137

Post by sdenton4 »

antonvs wrote: post #82
coppsilgold wrote: post #23

Earlier quoted context omitted.

Comparing Deep Learning with neuroscience may turn out to be erroneous. They may be orthogonal. The brain likely has more in common with Reservoir Computing (sans the actual learning algorithm) than Deep Learning. Deep Learning relies on end to end loss optimization, something which is much more powerful than anything the brain can be doing. But the end-to-end limitation is restricting, credit assignment is a big pro…

> If only we could train a model to just use Photoshop directly, but we can't. What kind of sadist would wish this on an intelligent entity?

Yeah, that's how you get skynet.

Re: There Will Be a Scientific Theory of Deep Learning

#138

Post by hodgehog11 »

energy123 wrote: post #128
hodgehog11 wrote: post #123

Earlier quoted context omitted.

Absolutely agree on both counts. Gradient boosting is the most commonly known and most successful variant, but it's the decision tree structure that is the underlying architecture there. Decision trees don't have the same "implicit training bias" phenomenon that neural networks have though, so all of this is just model bias in the classical statistical sense.

Can NNs be made to be better than trees on tabular data with some further constraints, or something?

You might find this interesting: https://arxiv.org/abs/1909.06312

Re: There Will Be a Scientific Theory of Deep Learning

#140

Post by tim333 »

Someone wrote: post #132
pash wrote: post #14

Earlier quoted context omitted.

The inflection point was 2012, when AlexNet [0], a deep convolutional neural net, achieved a step-change improvement in the ImageNet classification competition. After seeing AlexNet’s results, all of the major ML imaging labs switched to deep CNNs, and other approaches almost completely disappeared from SOTA imaging competitions. Over the next few years, deep neural networks took over in other ML domains as well. The…

> Even a mouse needs hundreds of millions of neurons to do what a mouse does. Under the very light assumption that a mouse doesn’t have neurons it doesn’t need, a mouse needs whatever number of neurons it has to do what a mouse does, so that’s not saying much. Reading https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n... , an ant has only 250k neurons and many reptiles can do with around 10 million. That pa…

Mice seem to have quite a good representation of the 3d environment around them and motor skills. I had one in my flat run off an jump through an approx 1 x 2 inch hole 6 inches off the ground and about 10 inches from where it jumped from. Humans would probably have a job with that and I've not seen a lizard say seem to have similar ability to know its way around.

I daresay I don't think animals actually need some number or neurons. There's probably just a trade off between more giving better results versus being heavier and more energy consuming.

Post reply on HN