Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There Will Be a Scientific Theory of Deep Learning

arxiv.org

Post reply on HN
Print view View original

61–70 of 179 posts

Re: There Will Be a Scientific Theory of Deep Learning

#61

Post by tacet »

mathisfun123 wrote: post #59
hodgehog11 wrote: post #39

As someone who works in the area, this provides a decent summary of the most popular research items. The most useful and impressive part is the set of open problems at the end, which just about covers all of the main research directions in the field. The skepticism I'm seeing in the comments really highlights how little of this work is trickling down to the public, which is very sad to see. While it can offer few mat…

> why do neural networks work better than other models The only people for whom this is an open question are the academics - everyone else understands it's entirely because of the bagillions of parameters.

Also massive human work done on them, that wasn't done before.

Data labeling is pretty big industry in some countries and I guess dropping 200 kilodollars on labeling is beyond the reach of most academics, even if they would not care about ethics of that.

Re: There Will Be a Scientific Theory of Deep Learning

#63

Post by jesseab »

cdavid wrote: post #57
pash wrote: post #14

Earlier quoted context omitted.

The inflection point was 2012, when AlexNet [0], a deep convolutional neural net, achieved a step-change improvement in the ImageNet classification competition. After seeing AlexNet’s results, all of the major ML imaging labs switched to deep CNNs, and other approaches almost completely disappeared from SOTA imaging competitions. Over the next few years, deep neural networks took over in other ML domains as well. The…

Indeed. I would add a third factor to compute and datasets: the lego-like aspect of NN that enabled scalable OSS DL frameworks. I did some ML in mid 2000s, and it was a PITA to reuse other people code (when available at all). You had some well known libraries for SVM, for HMM you had to use HTK that had a weird license, and otherwise looking at experiments required you to reimplement stuff yourself. Late 2000s had a…

Remember watching Alec Radford's Theano tutorial and feeling like I had found literal gold.

Re: There Will Be a Scientific Theory of Deep Learning

#65

Post by hodgehog11 »

mathisfun123 wrote: post #59
hodgehog11 wrote: post #39

As someone who works in the area, this provides a decent summary of the most popular research items. The most useful and impressive part is the set of open problems at the end, which just about covers all of the main research directions in the field. The skepticism I'm seeing in the comments really highlights how little of this work is trickling down to the public, which is very sad to see. While it can offer few mat…

> why do neural networks work better than other models The only people for whom this is an open question are the academics - everyone else understands it's entirely because of the bagillions of parameters.

No it isn't, and it's frustrating when the "common wisdom" tries to boil it down to this. If this was true, then the models with "infinitely many" parameters would be amazing. What about just training a gigantic two-layer network? There is a huge amount of work trying to engineer training procedures that work well.

The actual reason is due to complex biases that arise from the interaction of network architectures and the optimizers and persist in the regime where data scales proportionally to model size. The multiscale nature of the data induces neural scaling laws that enable better performance than any other class of models can hope to achieve.

Re: There Will Be a Scientific Theory of Deep Learning

#66

Post by jmalicki »

andbberger wrote: post #60
chadcmulligan wrote: post #53

Earlier quoted context omitted.

"why do neural networks work better than other models?" That sounds really interesting - any references (for a non specialist)?

https://en.wikipedia.org/wiki/Universal_approximation_theore... the better question is why does gradient descent work for them

The properties that the uniform approximation theorem proves are not unique to neural networks.

Any models using an infinite dimensional Hilbert space, such as SVMs with RBF or polynomial kernels, Gaussian process regression, gradient boosted decision trees, etc. have the same property (though proven via a different theorem of course).

So the universal approximation theorem tells us nothing about why should expect neural networks to perform better than those models.

Re: There Will Be a Scientific Theory of Deep Learning

#67

Post by cookiengineer »

hodgehog11 wrote: post #39

As someone who works in the area, this provides a decent summary of the most popular research items. The most useful and impressive part is the set of open problems at the end, which just about covers all of the main research directions in the field. The skepticism I'm seeing in the comments really highlights how little of this work is trickling down to the public, which is very sad to see. While it can offer few mat…

In my opinion current research should focus on revisiting older concepts to figure out if they can be applied to transformers.

Transformers are superior "database" encodings as the hype about LLMs points out, but there have been promising ML models that were focusing on memory parts for their niche use cases, which could be promising concepts if we could make them work with attention matrixes and/or use the frequency projection idea on their neuron weights.

The way RNNs evolved to LSTMs, GRUs, and eventually DNCs was pretty interesting to me. In my own implementations and use cases I wasn't able to reproduce Deepmind's claims in the DNC memory related parts. Back at the time the "seeking heads" idea of attention matrixes wasn't there yet, maybe there's a way to build better read/write/access/etc gates now.

[1] a fairly good implementation I found: https://github.com/joergfranke/ADNC

Re: There Will Be a Scientific Theory of Deep Learning

#68

Post by skydhash »

hodgehog11 wrote: post #65
mathisfun123 wrote: post #59

Earlier quoted context omitted.

> why do neural networks work better than other models The only people for whom this is an open question are the academics - everyone else understands it's entirely because of the bagillions of parameters.

No it isn't, and it's frustrating when the "common wisdom" tries to boil it down to this. If this was true, then the models with "infinitely many" parameters would be amazing. What about just training a gigantic two-layer network? There is a huge amount of work trying to engineer training procedures that work well. The actual reason is due to complex biases that arise from the interaction of network architectures and…

> The actual reason is due to complex biases that arise from the interaction of network architectures and the optimizers and persist in the regime where data scales proportionally to model size. The multiscale nature of the data induces neural scaling laws that enable better performance than any other class of models can hope to achieve.

That’s a lot of words to say that, if you encode a class of things as numbers, there’s a formula somewhere that can approximate an instance of that class. It works for linear regression and works as well for neural network. The key thing here is approximation.

Re: There Will Be a Scientific Theory of Deep Learning

#69

Post by fc417fc802 »

andbberger wrote: post #60
chadcmulligan wrote: post #53

Earlier quoted context omitted.

"why do neural networks work better than other models?" That sounds really interesting - any references (for a non specialist)?

https://en.wikipedia.org/wiki/Universal_approximation_theore... the better question is why does gradient descent work for them

I don't follow. Why wouldn't it work? It seems to me that a biased random walk down a gradient is about as universal as it gets. A bit like asking why walking uphill eventually results in you arriving at the top.

Re: There Will Be a Scientific Theory of Deep Learning

#70

Post by alasdair_ »

cdavid wrote: post #57
pash wrote: post #14

Earlier quoted context omitted.

The inflection point was 2012, when AlexNet [0], a deep convolutional neural net, achieved a step-change improvement in the ImageNet classification competition. After seeing AlexNet’s results, all of the major ML imaging labs switched to deep CNNs, and other approaches almost completely disappeared from SOTA imaging competitions. Over the next few years, deep neural networks took over in other ML domains as well. The…

Indeed. I would add a third factor to compute and datasets: the lego-like aspect of NN that enabled scalable OSS DL frameworks. I did some ML in mid 2000s, and it was a PITA to reuse other people code (when available at all). You had some well known libraries for SVM, for HMM you had to use HTK that had a weird license, and otherwise looking at experiments required you to reimplement stuff yourself. Late 2000s had a…

I think your post may have more acronyms than any other post I have ever read on hn. Do you have a guide to which specific things you are talking about with each acronym? Deep Learning and Machine Learning are obvious but some of the others I can’t follow at all - they could be so many different things.
Post reply on HN