Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There Will Be a Scientific Theory of Deep Learning

arxiv.org

Post reply on HN
Print view View original

171–179 of 179 posts

Re: There Will Be a Scientific Theory of Deep Learning

#171

Post by calf »

lou1306 wrote: post #119
calf wrote: post #114

Earlier quoted context omitted.

I'm not sure I agree with that. Even technically, my PC is not Turing-complete because its hard drive is finite. Yet there is an informal sense that Rice's Theorem is still relevant in a kind of PC abstraction sense, as we are all taught "virus checkers are strictly speaking impossible". This is a subtle point that needs further clarification from CS theorists, of which I am not. Neural networks in general are Turing…

But most "normal" neural networks are feed-forward, so they are guaranteed to terminate in a bounded amount of time. This rules Turing completeness right out. And even recurrent NNs can be "unfolded" into feed-forward equivalents, so they are not TC either. You need a memory element the network can interact with, just like an ALU by itself is not TC, but a barebones stateful CPU (ALU + registers) is.

I already addressed this type of misargument in my first paragraph. Another way of looking at it is, if NNs are so time bounded then they cannot be computationally powerful at all. Which is really strange.

Re: There Will Be a Scientific Theory of Deep Learning

#173

Post by Mithriil »

soVeryTired wrote: post #116
jmalicki wrote: post #66

Earlier quoted context omitted.

The properties that the uniform approximation theorem proves are not unique to neural networks. Any models using an infinite dimensional Hilbert space, such as SVMs with RBF or polynomial kernels, Gaussian process regression, gradient boosted decision trees, etc. have the same property (though proven via a different theorem of course). So the universal approximation theorem tells us nothing about why should expect ne…

Whenever people bring this up I like to remind them that linear interpolation is a universal function approximator.

I don't think that this is true. You need an infinite number of dimensions for this (think Taylor's expansion, Fourier expansion, infinitely wide or deep NNs..)

Re: There Will Be a Scientific Theory of Deep Learning

#174

Post by Mithriil »

NooneAtAll3 wrote: post #85
jmalicki wrote: post #66

Earlier quoted context omitted.

The properties that the uniform approximation theorem proves are not unique to neural networks. Any models using an infinite dimensional Hilbert space, such as SVMs with RBF or polynomial kernels, Gaussian process regression, gradient boosted decision trees, etc. have the same property (though proven via a different theorem of course). So the universal approximation theorem tells us nothing about why should expect ne…

Universal approximation is like saying that a problem is computable sure, that gives some relief - but it says nothing in practice unlike f.e. which side of P/NP divide the problem is on

Asymptotics has been used to validate tons of statistical tools. This is just another tool being validated.

If you have a tool that you don't know works when data increases (n-> infinity), then you shouldn't use it.

So practicaly, I believe it has serious implications.

Re: There Will Be a Scientific Theory of Deep Learning

#175

Post by Mithriil »

js8 wrote: post #115

Honestly, I found these two attempts at universal theory more interesting: https://arxiv.org/abs/2510.12269 https://www.mdpi.com/1099-4300/28/3/332 I am also interested in connection with fuzzy logic - it seems that NNs can reason in a fuzzy way, but what they are doing, formally? For years, people have been trying to formalize fuzzy reasoning but it looks like we don't care anymore. I feel like NNs (and transformers…

> nobody understands the fundamentals

Funny statement to be found in the discussion about... research results on the fundamentals.

Re: There Will Be a Scientific Theory of Deep Learning

#176

Post by jmalicki »

Mithriil wrote: post #173
soVeryTired wrote: post #116

Earlier quoted context omitted.

Whenever people bring this up I like to remind them that linear interpolation is a universal function approximator.

I don't think that this is true. You need an infinite number of dimensions for this (think Taylor's expansion, Fourier expansion, infinitely wide or deep NNs..)

Yes, you do linear interpolation between an infinite number of data points.

Re: There Will Be a Scientific Theory of Deep Learning

#177

Post by jmalicki »

Mithriil wrote: post #174
NooneAtAll3 wrote: post #85

Earlier quoted context omitted.

Universal approximation is like saying that a problem is computable sure, that gives some relief - but it says nothing in practice unlike f.e. which side of P/NP divide the problem is on

Asymptotics has been used to validate tons of statistical tools. This is just another tool being validated. If you have a tool that you don't know works when data increases (n-> infinity), then you shouldn't use it. So practicaly, I believe it has serious implications.

It's very much necessary but not sufficient. In real life the sample complexity matters a lot too, which is also asymptotics, but a more important one. E.g. how the central limit theorem is far more powerful than the law of large numbers.

Re: There Will Be a Scientific Theory of Deep Learning

#178

Post by antonvs »

krisoft wrote: post #145
r0ze-at-hn wrote: post #99

Earlier quoted context omitted.

We’re in a strange era where the Information-Theoretic foundations of deep learning are solidifying. The 'Why' is largely solved: it’s the efficient minimization of irreversible information loss relative to the noise floor. There is so much waste scaling models bigger and bigger when the math points to how to do it much more efficiently. One can take a great 70B model and have it run in only ~16GB with no loss in cap…

> One can take a great 70B model and have it run in only ~16GB with no loss in capability and the ability to keep training, but the last few years funding only went for "bigger". Awesome. What is holding you back? What do you need the funding for?

When an AI says things like that we call it “hallucination”.
Post reply on HN