Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Theory of Deep Learning

elonlit.com

Post reply on HN
Print view View original

51–60 of 68 posts

Re: A Theory of Deep Learning

#51

Post by yorwba »

vessenes wrote: post #46
yorwba wrote: post #45

Earlier quoted context omitted.

You don't need to understand that part of the derivation to implement it. You just need Algorithm 1 on page 33 of the paper. Or look at the author's implementation: https://github.com/elonlit/PopRiskMinimization/blob/main/pop...

Thanks for the link - I did not see a GitHub. So, your thoughts on the paper?

I think it's a solid theoretical contribution, but it might nonetheless fail to have practical relevance if some of their assumptions and approximations turn out to be too unrealistic. One way this could happen, for example, would be if typical training batches get gradients with a high-enough signal-to-noise ratio that their optimizer tweak ends up not tweaking much. Their somewhat unusual selection of experiments makes me suspect that this might be the case.

I read the paper earlier when it showed up on https://news.ycombinator.com/from?site=arxiv.org and the writing style of the blog post turned me off so I didn't bother to check how much it overhypes the results compared to the paper, but certainly a lot of people seem to have gotten the idea that this must be big if true, whereas I think it's better classified as neat, but not revolutionary.

Re: A Theory of Deep Learning

#52

Post by auggierose »

Looks like a typical machine learning paper to me. It cannot be understood unless you already kind of understand it. That is OK for communication with peers, but eventually I expect a "theory of" to be readable by anyone with a math degree.

Re: A Theory of Deep Learning

#53

Post by jeffrallen »

This looks like excellent work, it's reminding me of things I learned from Welch Labs vidoes. Given the amount of time I budget for keeping up with this stuff (regrettably too low) I'll wait until Welch Labs presents something on this.

Re: A Theory of Deep Learning

#55

Post by sigmoid10 »

scarmig wrote: post #48
sigmoid10 wrote: post #42

Earlier quoted context omitted.

Gradient descent is mathematically the most efficient optimization strategy (safe for some special functions) in high dimensions. This goes so far that people nowadays even believe it has to be used in the human brain [1], if only because every other method of updating the brain would be way too energy inefficient. From that perspective, finding the right parameterization was all we ever needed to achieve AI. [1] htt…

Even in supervised ML, pure gradient descent is not the most efficient optimization strategy. E.g., momentum is ubiquitous, and the updates it induces cannot be expressed as a gradient of some scalar loss. But the rotational non-gradient component of its updates substantially improves performance and convergence on the architectures we use. The brain probably primarily uses something like TD for task learning, which…

Pure gradient descent is not what happens in either field, but e.g. momentum is just another parameter constructed from historic gradients. While it is unlikely that the brain runs backpropagation the way you see it implemented in modern ML (same goes for TD btw), the core principle kind of needs to be the same from a pure large scale, high dimensional network efficiency POV. On top of that, adaptive plasticity is almost by definition about estimating useful directions of change. The key insight here would be that the brain does gradient estimation quite cheap and we can probably still learn a thing or two about modern ML from it.

Re: A Theory of Deep Learning

#56

Post by arolihas »

po1nt wrote: post #32
arolihas wrote: post #16

Idk to me this is just redescribing what deep neural networks do without actually explaining why anything happens. I guess it "unifies" things but I am kinda over most unifying theories. Everything is Bayesian, everything is a graph or a group or some other fancy geometric structure, everything is a category. Ultimately the best framework is whatever is useful enough to explain what's happening in such a way that a p…

We're still in the era of room-sized-computers-only-scientists-understand era of the neural networks. Knobs and buttons for nerds are slowly coming.

I agree, which is why it’s too early to make such grandiose claims about deep learning theory.

Re: A Theory of Deep Learning

#57

Post by airza »

piskov wrote: post #13
airza wrote: post #4

A very fascinating read. As a fellow tufte css enjoyer, Why is user select turned off on the sidenotes? I would like to be able to copy paste them quite badly.

Layout is fine but font is atrocious. Uppercase letters have different stroke width than lowercase ones — it’s like they are *B*old *L*ike this. Not only that: tracking, kerning is basically non-existent. Please don’t use that open-source font You need real Bembo, not that piece of shit

Are you on Windows? I patched the original tufte fonts to fix these crazy kerning issues, but i have indeed noticed them.

Re: A Theory of Deep Learning

#58

Post by sdenton4 »

sigmoid10 wrote: post #42
hellohello2 wrote: post #29

Earlier quoted context omitted.

I agree, this is the correct way to see it IMO. Instead of designing better optimizers, we designed easier parameterizations to optimize. The surprising part is that these parameterizations exist in the first place.

Gradient descent is mathematically the most efficient optimization strategy (safe for some special functions) in high dimensions. This goes so far that people nowadays even believe it has to be used in the human brain [1], if only because every other method of updating the brain would be way too energy inefficient. From that perspective, finding the right parameterization was all we ever needed to achieve AI. [1] htt…

Taking a quick look at the paper...

Their claim isn't that the brain uses gradient descent, but that the direction of updates has (on average) positive inner product with the gradient. I expect this would also be true for (say) simulated annealing, yet we don't say that simulated annealing is gradient descent.

There's also a discussion of loss functions and how they relate to the update missing - as far as I know, there's still no great notion of how the brain picks a global loss function, and no mechanism for backprop. In this paper, looking at a specific learning task you can define a loss function extrinsically allowing us to talk about the gradient, but how that relates to things happening in the brain is a big big mystery.

Re: A Theory of Deep Learning

#59

Post by jhanschoo »

My intuitive understanding about double descent is that

1. Older ML models encoded in their architecture and lack of expressivity a bias to simplicity; which aided interpolation.

2. Overparameterized models instead use regularization to nudge parameters to simpler and more robust representations, while still memorizing the noise. In this manner, we still achieve generalization performance OOD. Moreover, the softer nudging and fundamental architectural expressivity allows for "data-specific" generalizations and representations that may be impossible to represent in small models. 3. At the critical point between the two regimes, the model is expressive enough to memorize; but not expressive enough to simultaneously both do that and encode general patterns.

I wonder how this understanding translates to these researchers' models of deep learning.

Post reply on HN