Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Theory of Deep Learning

elonlit.com

Post reply on HN
Print view View original

31–40 of 68 posts

Re: A Theory of Deep Learning

#31

Post by ipnon »

arolihas wrote: post #16

Idk to me this is just redescribing what deep neural networks do without actually explaining why anything happens. I guess it "unifies" things but I am kinda over most unifying theories. Everything is Bayesian, everything is a graph or a group or some other fancy geometric structure, everything is a category. Ultimately the best framework is whatever is useful enough to explain what's happening in such a way that a p…

A real theory would predict phenomena thus far unseen. We already know about this 4 part taxonomy.

Re: A Theory of Deep Learning

#32

Post by po1nt »

arolihas wrote: post #16

Idk to me this is just redescribing what deep neural networks do without actually explaining why anything happens. I guess it "unifies" things but I am kinda over most unifying theories. Everything is Bayesian, everything is a graph or a group or some other fancy geometric structure, everything is a category. Ultimately the best framework is whatever is useful enough to explain what's happening in such a way that a p…

We're still in the era of room-sized-computers-only-scientists-understand era of the neural networks. Knobs and buttons for nerds are slowly coming.

Re: A Theory of Deep Learning

#33

Post by pixelpoet »

ks2048 wrote: post #11

The relevant paper: "A Theory of Generalization in Deep Learning". https://arxiv.org/abs/2605.01172

I interpreted the kernel K of this paper as the BRDF in Rendering Equation[0] and its familiar diffusion process (from light transport simulation, or really any integro-differential equation system); together with https://en.wikipedia.org/wiki/Neural_tangent_kernel I hope this paper might be accessible with some study

[0] https://en.wikipedia.org/wiki/Rendering_equation

Re: A Theory of Deep Learning

#35

Post by codemog »

Where’s the theory of how the human brain does what it does? Maybe these high dimensional structures don’t have a nice compact “theory”. Trying to fit these systems into a nice compact theory is a very human thing, but not everything works like that.

Re: A Theory of Deep Learning

#36

Post by hashmap »

this landed precisely on like 3 weird bugs ive been hitting and solving in different stupid ways for dealing with things like sgd collapsing too many good answers into one bad answer, and gave me a real direction to try to fix the link missing in my own ml stuff. what timing. i have tried analytic solutions too and they're useful for like mapping prompts into memory geometry but from there ive ended up still having to use sgd. cause i think what happens is, sgd teaches the neural net both the geometry and how to navigate it. if you just teleport to the answer it doesnt learn how to walk.

Re: A Theory of Deep Learning

#37

Post by jldugger »

prideout wrote: post #5

This is a fascinating mathematical framework, but the post title might be a bit of an overreach. I often wonder if "a theory of deep learning" could exist that could be stated succinctly and that could predict (1) scaling laws and (2) the surprising reliability of gradient descent. Note that I said "predict" not "describe". It feels like we're still in the era of Kepler, not Newton.

[flagged]

Re: A Theory of Deep Learning

#38

Post by 317070 »

dwrodri wrote: post #3
refulgentis wrote: post #2

This is a beautifully written way of saying “Some parts of what the network memorizes affect test behavior, and some don’t.” But that’s not a theory of deep learning, the grand unified theory would explain that. We're given a signal channel and a reservoir . Signal lives in the channel, noise lives in the reservoir, and the reservoir supposedly doesn’t show up at test time. Okay, but then we have: why would SGD put t…

Admittedly probably some aggrandized boasting here, but I think empirical verification of that Adam modification alone would be a meaningful contribution, unless that's prior work?

A theory that skips the parameter space, and understands grokking theory, comes up with an unexplained update rule, which notably works on a per-parameter level by dropping the updates for most parameters.

I suspect there is going to be a lot of handwaving to actually go from eNTK to that new update rule.

I also doubt it helps in the non-grokking regime, given the focus of the theory, which is where all the practical applications I have ever heard from live.

Don't get me wrong, I did enjoy reading this essay. It's well written and reasonably argumented without going into details.

Re: A Theory of Deep Learning

#39

Post by renticulous »

xiaodai wrote: post #23

What a beautifully written article. It's extremely that I favourite an article but this is one.

The Hidden Physics of LLMs: Retrieval as Thermodynamics

https://www.youtube.com/watch?v=ppCZfjLdSY8

I found this video to be illustrative as well. Simple and anyone can understand.

Re: A Theory of Deep Learning

#40

Post by SubiculumCode »

refulgentis wrote: post #2

This is a beautifully written way of saying “Some parts of what the network memorizes affect test behavior, and some don’t.” But that’s not a theory of deep learning, the grand unified theory would explain that. We're given a signal channel and a reservoir . Signal lives in the channel, noise lives in the reservoir, and the reservoir supposedly doesn’t show up at test time. Okay, but then we have: why would SGD put t…

I don't know the math, but this point was clear to me and it screamed, "crank" but not being sure of that because I am not learned enough to understand the math... but even I could tell the magnitude of the claim. Even just the removing the need for validation sets would have epic consequences across many fields.
Post reply on HN