Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Theory of Deep Learning

elonlit.com

Post reply on HN
Print view View original

41–50 of 68 posts

Re: A Theory of Deep Learning

#41

Post by yorwba »

minimaltom wrote: post #25

> That is, if the batch signal on a parameter exceeds its leave-one-out noise, update it; if not, skip it. This is a one-line change to Adam that accelerates grokking by 5x, suppresses memorization in PINNs, and improves DPO fine-tuning, eliminating the need for validation sets entirely. Does anyone understand the formula they expressed above this sentence? is this just the classic "skip updating parameters with high…

What is classic about "skip updating parameters with high gradient/loss variance in multiple batches/samples"? Do you have a particular algorithm in mind that uses this heuristic?

Re: A Theory of Deep Learning

#42

Post by sigmoid10 »

hellohello2 wrote: post #29
sdenton4 wrote: post #27

Earlier quoted context omitted.

I dunno... gradient descent is only really reliable with a big bag of tricks. Knowing good initializations is a starting point, but recurrent connections and batch/layer normalization go a very long way towards making it reliable.

I agree, this is the correct way to see it IMO. Instead of designing better optimizers, we designed easier parameterizations to optimize. The surprising part is that these parameterizations exist in the first place.

Gradient descent is mathematically the most efficient optimization strategy (safe for some special functions) in high dimensions. This goes so far that people nowadays even believe it has to be used in the human brain [1], if only because every other method of updating the brain would be way too energy inefficient. From that perspective, finding the right parameterization was all we ever needed to achieve AI.

[1] https://physoc.onlinelibrary.wiley.com/doi/full/10.1113/JP28...

Re: A Theory of Deep Learning

#43

Post by yorwba »

hashta wrote: post #21

Interesting read. I remember the grokking paper when it came out but I don't think I've ever seen that classic grokking loss curve in my own hands on real data. Curious if others have seen it more often in practice

To get pure grokking, you need a model large enough to easily memorize the entire training data and keep training for a long time after memorization. In practice, you'll probably use a more realistically-sized model that might grok on some subset of the data, but not so strongly that it's extremely obvious.

Re: A Theory of Deep Learning

#44

Post by yorwba »

317070 wrote: post #38
dwrodri wrote: post #3

Earlier quoted context omitted.

Admittedly probably some aggrandized boasting here, but I think empirical verification of that Adam modification alone would be a meaningful contribution, unless that's prior work?

A theory that skips the parameter space, and understands grokking theory, comes up with an unexplained update rule, which notably works on a per-parameter level by dropping the updates for most parameters. I suspect there is going to be a lot of handwaving to actually go from eNTK to that new update rule. I also doubt it helps in the non-grokking regime, given the focus of the theory, which is where all the practical…

The handwaving required is just to assume a diagonal preconditioner, and the optimal preconditioner under that constraint corresponds to the new update rule. (See section F of the paper.) And of course a diagonal preconditioner works on the per-paramer level.

Re: A Theory of Deep Learning

#45

Post by yorwba »

vessenes wrote: post #30

So, this is either the paper of the year, or ... definitely not the paper of the year. https://arxiv.org/pdf/2605.01172 is the current version. The money graphs are page 8 and on where they show (some weirdly thick) line charts with loss results reached in roughly 1/5 the number of steps that Adam takes, just what the blog post mentions. They also claim holding back test data is not needed, also with more graphs. I'm…

You don't need to understand that part of the derivation to implement it. You just need Algorithm 1 on page 33 of the paper. Or look at the author's implementation: https://github.com/elonlit/PopRiskMinimization/blob/main/pop...

Re: A Theory of Deep Learning

#46

Post by vessenes »

yorwba wrote: post #45
vessenes wrote: post #30

So, this is either the paper of the year, or ... definitely not the paper of the year. https://arxiv.org/pdf/2605.01172 is the current version. The money graphs are page 8 and on where they show (some weirdly thick) line charts with loss results reached in roughly 1/5 the number of steps that Adam takes, just what the blog post mentions. They also claim holding back test data is not needed, also with more graphs. I'm…

You don't need to understand that part of the derivation to implement it. You just need Algorithm 1 on page 33 of the paper. Or look at the author's implementation: https://github.com/elonlit/PopRiskMinimization/blob/main/pop...

Thanks for the link - I did not see a GitHub.

So, your thoughts on the paper?

Re: A Theory of Deep Learning

#47

Post by r0ze-at-hn »

Linking to the paper: https://arxiv.org/pdf/2605.01172 which is also a fantastic read, the application to deep learning is good. It does a lot of cross-mapping and highlighting a bunch of old stuff that is named differently in this paper and worth calling out for those with those backgrounds:

"Cumulative Dissipation Gramian" Ws = Observability Gramian (from Control Theory). For example the spectral cutoff is exactly the Hankel singular value truncation from model reduction.

"Signal Channel" / "Reservoir" is Controllable/Observable vs. Uncontrollable/Unobservable Subspaces. Using Adamjan-Arov-Krein (AAK) theory gives the optimal nonlinear reduced model answering the optimal compression question.

"Drift–Diffusion Separation" is Freidlin-Wentzell Large Deviation Theory. They can predict "grokking" time from the FW action.

"Population-Risk Gate" is Quantum Weak Value / Postselection (Aharonov)

So for the follow-up problems

Control theory gives the truncation error bounds for model compression. Large deviation theory gives the grokking time predictions. Quantum measurement theory gives the imaginary preconditioners. Information geometry gives the optimal continuous relaxation of the gate.

Some nice implications of new ways of doing stuff which are nice to see formalized here:

Old: Pick architecture, hope it generalizes New: Design architecture to maximize observability Gramian rank (Honestly we pull a lot from control theory here)

Old: Use validation set to detect overfitting New: Monitor λ(Ws) spectrum during training; no validation needed

Old: Prune post-hoc based on magnitude New: Prune during training based on ker(Ws) membership

Old: Fixed learning rate New: Spectral learning rate

Re: A Theory of Deep Learning

#48

Post by scarmig »

sigmoid10 wrote: post #42
hellohello2 wrote: post #29

Earlier quoted context omitted.

I agree, this is the correct way to see it IMO. Instead of designing better optimizers, we designed easier parameterizations to optimize. The surprising part is that these parameterizations exist in the first place.

Gradient descent is mathematically the most efficient optimization strategy (safe for some special functions) in high dimensions. This goes so far that people nowadays even believe it has to be used in the human brain [1], if only because every other method of updating the brain would be way too energy inefficient. From that perspective, finding the right parameterization was all we ever needed to achieve AI. [1] htt…

Even in supervised ML, pure gradient descent is not the most efficient optimization strategy. E.g., momentum is ubiquitous, and the updates it induces cannot be expressed as a gradient of some scalar loss. But the rotational non-gradient component of its updates substantially improves performance and convergence on the architectures we use.

The brain probably primarily uses something like TD for task learning, which is also not expressible as a gradient of any objective function. And, though the paper mentions Hebbian learning, it's only very particular network architectures (e.g. single neuron; symmetric connections) that you can treat its updates as a gradient of some energy function; these architectures aren't anything close to what we see in the brain.

Re: A Theory of Deep Learning

#49

Post by T-A »

ipnon wrote: post #31
arolihas wrote: post #16

Idk to me this is just redescribing what deep neural networks do without actually explaining why anything happens. I guess it "unifies" things but I am kinda over most unifying theories. Everything is Bayesian, everything is a graph or a group or some other fancy geometric structure, everything is a category. Ultimately the best framework is whatever is useful enough to explain what's happening in such a way that a p…

A real theory would predict phenomena thus far unseen. We already know about this 4 part taxonomy.

Did you also know about this?

Lastly, we derive an exact population-risk objective from a single training run with no validation data, for any architecture, loss, or optimizer, and prove that it measures precisely the noise in the signal channel. This objective reduces in practice to an SNR preconditioner on top of Adam, adding one state vector at no extra cost; it accelerates grokking by 5x, suppresses memorization in PINNs and implicit neural representations, and improves DPO fine-tuning under noisy preferences while staying 3x closer to the reference policy. [1]

[1] https://arxiv.org/abs/2605.01172

Re: A Theory of Deep Learning

#50

Post by kleiba2 »

> This exact characterization is possible because in output space, training dynamics can be understood through a locally linear differential equation along the realized path, where dominant eigenmodes of the evolving kernel equilibrate exponentially fast. Forcing an optimizer to slowly step through these solved directions is highly inefficient and suggests a path to analytically jump to the final network state.

But at what computational cost?

Post reply on HN