Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Theory of Deep Learning

elonlit.com

Post reply on HN
Print view View original

61–68 of 68 posts

Re: A Theory of Deep Learning

#61

Post by vessenes »

yorwba wrote: post #51
vessenes wrote: post #46

Earlier quoted context omitted.

Thanks for the link - I did not see a GitHub. So, your thoughts on the paper?

I think it's a solid theoretical contribution, but it might nonetheless fail to have practical relevance if some of their assumptions and approximations turn out to be too unrealistic. One way this could happen, for example, would be if typical training batches get gradients with a high-enough signal-to-noise ratio that their optimizer tweak ends up not tweaking much. Their somewhat unusual selection of experiments m…

Thanks. I found the DPO test very interesting where they intentionally twiddled the dataset to create 'noise'. That would I think back your concern. But on the other hand a method that allows noisier data in or does better with data sets that are low signal across bands would be nice to have.

Re: A Theory of Deep Learning

#62

Post by chermi »

refulgentis wrote: post #2

This is a beautifully written way of saying “Some parts of what the network memorizes affect test behavior, and some don’t.” But that’s not a theory of deep learning, the grand unified theory would explain that. We're given a signal channel and a reservoir . Signal lives in the channel, noise lives in the reservoir, and the reservoir supposedly doesn’t show up at test time. Okay, but then we have: why would SGD put t…

I find the landscape perspective very valuable when trying to understand NN. Why SGD finds the right buckets? I am not super current, but this looked like the right trail to me -- https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.12...

Re: A Theory of Deep Learning

#63

Post by hashta »

yorwba wrote: post #43
hashta wrote: post #21

Interesting read. I remember the grokking paper when it came out but I don't think I've ever seen that classic grokking loss curve in my own hands on real data. Curious if others have seen it more often in practice

To get pure grokking, you need a model large enough to easily memorize the entire training data and keep training for a long time after memorization. In practice, you'll probably use a more realistically-sized model that might grok on some subset of the data, but not so strongly that it's extremely obvious.

I think I trained models with #params >> #training examples for hundreds of epochs, but still don't recall seeing that loss curve on real data. Curious if others have seen it with larger models or much longer runs

Re: A Theory of Deep Learning

#64

Post by minimaltom »

yorwba wrote: post #41
minimaltom wrote: post #25

> That is, if the batch signal on a parameter exceeds its leave-one-out noise, update it; if not, skip it. This is a one-line change to Adam that accelerates grokking by 5x, suppresses memorization in PINNs, and improves DPO fine-tuning, eliminating the need for validation sets entirely. Does anyone understand the formula they expressed above this sentence? is this just the classic "skip updating parameters with high…

What is classic about "skip updating parameters with high gradient/loss variance in multiple batches/samples"? Do you have a particular algorithm in mind that uses this heuristic?

Theres been multiple papers discussing how only updating parameters that have high agreement in update direction leads to less overfitting and better generalization. Lemme see if I can find em.

https://arxiv.org/abs/2411.16085 - set updates to 0 where theres disagreement in the sign of the parameter update - got accepted!

https://arxiv.org/pdf/2412.18052 - discard gradient updates from batches/minibatches that disagree where disagree means cosine distance threshold (they solved for 0.97 or something being optimal)

Re: A Theory of Deep Learning

#65

Post by johnthescott »

r0ze-at-hn wrote: post #47

Linking to the paper: https://arxiv.org/pdf/2605.01172 which is also a fantastic read, the application to deep learning is good. It does a lot of cross-mapping and highlighting a bunch of old stuff that is named differently in this paper and worth calling out for those with those backgrounds: "Cumulative Dissipation Gramian" Ws = Observability Gramian (from Control Theory). For example the spectral cutoff is exactly…

> We present a non-asymptotic theory of generalization

what is a non-elephant animal (to paraphrase stan ulam)?

Re: A Theory of Deep Learning

#66

Post by hellohello2 »

sigmoid10 wrote: post #42
hellohello2 wrote: post #29

Earlier quoted context omitted.

I agree, this is the correct way to see it IMO. Instead of designing better optimizers, we designed easier parameterizations to optimize. The surprising part is that these parameterizations exist in the first place.

Gradient descent is mathematically the most efficient optimization strategy (safe for some special functions) in high dimensions. This goes so far that people nowadays even believe it has to be used in the human brain [1], if only because every other method of updating the brain would be way too energy inefficient. From that perspective, finding the right parameterization was all we ever needed to achieve AI. [1] htt…

Hmm I'm not sure what you mean by "Gradient descent is mathematically the most efficient optimization strategy". Do you mean that gradient-based optimization in general? (in other words do you consider Adam gradient descent?)

Re: A Theory of Deep Learning

#67

Post by hellohello2 »

sdenton4 wrote: post #58
sigmoid10 wrote: post #42

Earlier quoted context omitted.

Gradient descent is mathematically the most efficient optimization strategy (safe for some special functions) in high dimensions. This goes so far that people nowadays even believe it has to be used in the human brain [1], if only because every other method of updating the brain would be way too energy inefficient. From that perspective, finding the right parameterization was all we ever needed to achieve AI. [1] htt…

Taking a quick look at the paper... Their claim isn't that the brain uses gradient descent, but that the direction of updates has (on average) positive inner product with the gradient. I expect this would also be true for (say) simulated annealing, yet we don't say that simulated annealing is gradient descent. There's also a discussion of loss functions and how they relate to the update missing - as far as I know, th…

Why would this be true for simulated annealing?

Re: A Theory of Deep Learning

#68

Post by sdenton4 »

hellohello2 wrote: post #67
sdenton4 wrote: post #58

Earlier quoted context omitted.

Taking a quick look at the paper... Their claim isn't that the brain uses gradient descent, but that the direction of updates has (on average) positive inner product with the gradient. I expect this would also be true for (say) simulated annealing, yet we don't say that simulated annealing is gradient descent. There's also a discussion of loss functions and how they relate to the update missing - as far as I know, th…

Why would this be true for simulated annealing?

Because it improves the loss!

The gradient is the direction in which loss improves the fastest. Moving in a direction with a positive dot product with the gradient just means that you're (locally) improving the loss.

Post reply on HN