Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There Will Be a Scientific Theory of Deep Learning

arxiv.org

Post reply on HN
Print view View original

111–120 of 179 posts

Re: There Will Be a Scientific Theory of Deep Learning

#111

Post by bobbruno »

porcoda wrote: post #20
RyanShook wrote: post #5

Here's where I'm missing understanding: for decades the idea of neural networks had existed with minimal attention. Then in 2017 Attention Is All You Need gets released and since then there is an exponential explosion in deep learning. I understand that deep learning is accelerated by GPUs but the concept of a transformer could have been used on much slower hardware much earlier.

As others pointed out, the explosion of interest started with the deep convolutional networks that were applied in image problems. What I always thought was interesting was that prior to that, NNs were largely dismissed as interesting. When I took a course on them around the year 2000 that was the attitude most people took. It seems like what it took to spark renewed interest was ImageNet and seeing what you get when…

I played with NNs in the late 80's/early 90s, with little more than a copy of Hinton's paper, a PC and a C compiler. Obviously, I got no practical results. But I got the intuition of how they worked and what they could potentially do.

Cut to 2008-9,and I started to see smartphones, grid (then cloud) computing and social networks emerging. My MBA dissertation, finished in 2011, was about how that would change the world, because the requirements for meaningful AI were coming along - data and compute. The theory was already there, Hinton, LeCun, Schmidhuber,etc.

That got me back into the Data Science field, after years working in Data Engineering. Too bad I lived in Brazil back then and couldn't find a way to join the emerging scene in California and other top places. I'd be rich now...

Re: There Will Be a Scientific Theory of Deep Learning

#112

Post by hackinthebochs »

jimmypk wrote: post #108
hodgehog11 wrote: post #73

Earlier quoted context omitted.

Extremely well said. Universal approximation is necessary but not sufficient for the performance we are seeing. The secret sauce is implicit regularization, which comes about analogously to enforcing compression.

[flagged]

>Do you think grokking is consistent with implicit regularization as compression

Pretty sure it's been shown that grokking requires L1 regularization which pushes model parameters towards zero. This can be viewed as compression in the sense of encoding the distribution in the fewest bits possible, which happens to correspond to better generalization.

Re: There Will Be a Scientific Theory of Deep Learning

#113

Post by charcircuit »

imtringued wrote: post #107
charcircuit wrote: post #88

Earlier quoted context omitted.

>you'd need every single one of them, millions up millions of them, to be all zero If they were all correlated with each other that does not seem far fetched.

Ok but it's already known that you shouldn't initialize your network parameters to a single constant and instead initialize the parameters with random numbers.

The model can converge towards such a state even if randomly initialized.

Re: There Will Be a Scientific Theory of Deep Learning

#114

Post by calf »

sanxiyn wrote: post #103
calf wrote: post #52

Is there not some Rice's Theorem equivalent for deep nets? After all they are machines that are randomly generated, so from classical computer science I would not presume a theory of "what do all deep nets do" to be prima facie logically possible. Nor do I see this explained in the objections section.

As I understand, Rice's theorem does not apply because neural networks are not Turing-complete.

I'm not sure I agree with that. Even technically, my PC is not Turing-complete because its hard drive is finite. Yet there is an informal sense that Rice's Theorem is still relevant in a kind of PC abstraction sense, as we are all taught "virus checkers are strictly speaking impossible". This is a subtle point that needs further clarification from CS theorists, of which I am not.

Neural networks in general are Turing models. Human brains are in the abstract Turing complete as well, as a simple example. LLMs being run iteratively in an unbounded loop may be "effectively Turing complete" for this simple reason, as well.

Regardless, any theory purporting to be foundational ought to explicitly address this demarcation. Unless practitioners think computability and formal complexity are not scientific foundations for CS.

Re: There Will Be a Scientific Theory of Deep Learning

#115

Post by js8 »

Honestly, I found these two attempts at universal theory more interesting:

https://arxiv.org/abs/2510.12269

https://www.mdpi.com/1099-4300/28/3/332

I am also interested in connection with fuzzy logic - it seems that NNs can reason in a fuzzy way, but what they are doing, formally? For years, people have been trying to formalize fuzzy reasoning but it looks like we don't care anymore.

I feel like NNs (and transformers) are the OOP (object-oriented programming) of ML. Really popular, works pretty well in practice, but nobody understands the fundamentals; there is a feeling it is a made up new language to express things expressible before, but hard to pinpoint where exactly it helps.

Re: There Will Be a Scientific Theory of Deep Learning

#116

Post by soVeryTired »

jmalicki wrote: post #66
andbberger wrote: post #60

Earlier quoted context omitted.

https://en.wikipedia.org/wiki/Universal_approximation_theore... the better question is why does gradient descent work for them

The properties that the uniform approximation theorem proves are not unique to neural networks. Any models using an infinite dimensional Hilbert space, such as SVMs with RBF or polynomial kernels, Gaussian process regression, gradient boosted decision trees, etc. have the same property (though proven via a different theorem of course). So the universal approximation theorem tells us nothing about why should expect ne…

Whenever people bring this up I like to remind them that linear interpolation is a universal function approximator.

Re: There Will Be a Scientific Theory of Deep Learning

#117

Post by olalonde »

NitpickLawyer wrote: post #8
4b11b4 wrote: post #4

wow.. this would be cool. Instead of just.. guessing "shapes"

tbf, we've learned (ha!) more from smashing teeny tiny particles and "looking" at what comes out than from say 40 years of string theory. Sometimes doing stuff works, and the theory (hopefully) follows.

Same with electricity. We had Ohm's Law and were building electrical devices (e.g. telegraph, lightbulb) long before we discovered the electron.

Re: There Will Be a Scientific Theory of Deep Learning

#119

Post by lou1306 »

calf wrote: post #114
sanxiyn wrote: post #103

Earlier quoted context omitted.

As I understand, Rice's theorem does not apply because neural networks are not Turing-complete.

I'm not sure I agree with that. Even technically, my PC is not Turing-complete because its hard drive is finite. Yet there is an informal sense that Rice's Theorem is still relevant in a kind of PC abstraction sense, as we are all taught "virus checkers are strictly speaking impossible". This is a subtle point that needs further clarification from CS theorists, of which I am not. Neural networks in general are Turing…

But most "normal" neural networks are feed-forward, so they are guaranteed to terminate in a bounded amount of time. This rules Turing completeness right out. And even recurrent NNs can be "unfolded" into feed-forward equivalents, so they are not TC either.

You need a memory element the network can interact with, just like an ALU by itself is not TC, but a barebones stateful CPU (ALU + registers) is.

Post reply on HN