Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

So this is a book written by applied mathematicians for applied mathematics (they state in the preface it’s for scientists, but some theoretical scientists and engineers are essentially applied mathematics). As a result, both the topics and the presentation are biased towards those types of people. For example, I’ve never seen in practice worry about the existence and uniqueness conditions for their gradient-based optimization algorithm in deep learning. However, that’s the kind of result those people do care about and academic papers are written on the topic. The title does say that this is a book on the theoretical underpinnings of the subject, so I am not surprised that it is written this way. People also don’t necessarily read these books cover-to-cover, but drill into the few chapters that use techniques relevant to what they themselves are researching. There was a similarly verbose monograph I used to use in my research, but only about 20-30 pages had the meat I was interested in.

This kind of book is more verbose than my liking both in terms of rigor and content. For example, they include Gronwall’s inequality as a lemma and prove it. The version that they use is a bit more general than the one I normally see, but Gronwall’s inequality is a very standard tool in analyzing ODEs and I have rigorous control theory books that state it without proof to avoid clutter (they do provide a reference to a proof). A lot of this verbosity comes about when your standard of proof is high and the assumptions you make are small.

A little bit of A and B. You can do a lot with very little math beyond linear algebra, calculus, and undergraduate probability, and that knowledge is mainly there to provide intuition and formalize the problem that you’re solving a bit. You also churn out results (including very impressive ones) without doing any math.

A result of the above is that people are empirically demonstrating new problems and solving them very quickly — much more quickly than people can come up with theoretical results explaining why they work. The theory is harder to come by for a few reasons, but many of the successful examples of deep learning don’t fit nicely into older frameworks from, e.g., statistics and optimal control, to explain them well.

> "the scaling hypothesis"

Oh boy, don't get me started.... I first off should say that by no means do I think any of these people (at least those publishing) are dumb. You can also be a genius in one direction and a fucking idiot in another, and that's okay. Certainly describes me haha (well less on the genius side and more on the functioning idiot side. So take everything I say with a grain of salt). Don't get me wrong, scale is incredibly important and is certainly the reason for our recent advancements. But scale taking us to AGI is fairly naive to me. The idea here has a few clear assumptions being made. First is that the data can accurately explain all phenomena if the machine is capable of sufficient imputation. I just don't even know how to tackle this one because it is so well established as false in the statistics literature. Another is that RLHF is enough for alignment. I like to say that RLHF is like Justice Stewart's definition of porn: I know it when I see it. This is certainly a useful tool, but we shouldn't be naive about its limitations. Just go on any reddit discussion on what constitutes NSFW and you'll find tons of disagreement or even the HN discussions of "Is This A Vehicle"[0]. Those comments are just beautiful and crazygringo (top comment) demonstrates this all perfectly. There's a powerful inference and imputation game going hand in hand and this is the issue. There needs to be more time spent thinking about one's brain and questioning assumptions we've made. As you advance, details become more and more important. We get tricked because you can often get away without nuance in the beginning of studying something but with sufficient expertise nuance ends up dominating the discussion and you might often actually see that naivety doesn't take a step in the right direction but rather can take you a step in the wrong direction (but often moving is more important). I'll reference Judea Pearl and Ilya on this one[1]. Pearl is absolutely correct, even if not conveyed well (it is Twitter after all). His book will give a good understanding of this though.

> What math would you use to describe the limitations of deep learning?

This is hard, because there isn't as much research in it as there is in demonstrations. I wouldn't go as far as saying that there's no work, but it is just far less popular and advancements are slower. Some optimal transport people really get into this stuff as well as people that work on Normalizing Flows. Aapo Hyvarinen is a really good person to read and you'll find foundations for many things like diffusion in his works that far predate the boom. I'd also really suggest looking at Max Welling and any/all of his students. If you go down that path you'll find many more people but this is a good place to enter that network.

But honestly, the best math to get started on to learn this stuff isn't "ML math". It's statistics, probability, metric theory, topology, linear algebra, and many specialized domains within these. I'd even go as far to say that category theory and set theory are very useful. It's all that math that you learn for a lot of other things, but you just need to have the correct lens. There is a problem in math education that we're often either far too application focused or too abstract focused that we forget to be generalist and have that deeper understanding[2]. But this is a lot and I'm not sure of a good single resource that pulls it all together in a way good for introductions (this paper certainly has many of the things I'd mention but it is not introductory). After all, things are simpler after they are understood.

I've written a lot and feel like I may have not given a sufficient answer. There's a lot to say and it is hard to convey in general language to general audiences. But I think I have given enough to find the path you're asking about but just wouldn't suggest you're going to get a complete answer in a comment, unfortunately (maybe someone is a better communicator than me)

[0] https://news.ycombinator.com/item?id=36453856

[1] https://twitter.com/yudapearl/status/1735211875191910550

[2] I think the theory focused people do often understand this more, but that's usually after going through the gauntlet and likely isn't even seen by them along that journey and especially prior to the point where many people stop. Certainly Terry Tao understands how math is just models and something like "the wave equation" isn't specifically about waves and far more general. You'll also find a lot of breaktroughs where the key ingredient is taking something from one domain and shoving it into another. Patchwork is often needed but sometimes it gets more generalized (or they derive a generalization, then show that the two are specific instances of that general form).

> The theorems are just hints about what might be happening.

Isn't this true everywhere? Certainly it is just, in the words of Asimov, the relativity of wrongness. I mean even physics is "just a hint" despite being an incredibly strong one. I think maybe a lot of people might not agree but I think a lot of people aren't as aware of all the research that still goes on in every day physics. Like studying ocean waves/currents, wind, explosions, materials, and so much more that is not quantum or relativity. But quantum and relativity get far more attention, so perception bias.

> The key deep learning knowledge is in papers that basically only show that X approach works best on Y (plus maybe some suggestive theorem)

I very much disagree. Those are certainly the most visible, but not the most foundational. Actually I believe this approach is holding us back, and diffusion is my best example of this. Big steps in diffusion and GANs were made around the same time, but GANs were easier to implement and less resource heavy. Sohl-Dickstein certainly is a key player despite Ho being more well known. Same with Aapo Hyvarinen. I think we got too captivated by GANs that it made it harder to publish anything else. I've had some experience with this personally, where I've given up trying to publish in Normalizing Flows because reviewers will ask why my works are not better than GANs (or now diffusion) despite being better than other Flows or even got this on a distillation based paper (multiple times before we abandoned it). If there's too heavy of concern on metrics (not using as guides/hints, but as targets) then how can other things advance in a normal way? You'd have to take leaps and bounds instead of incrementalism (which we've established is fine for popular paths. Former GANs, now diffusion). Leaps and bounds because the community size is exceptionally disproportionate and that is far more time and research being put into one than another. I'd argue that we have pretty good evidence to believe a hypothesis that counterfactually diffusion would have emerged as a strong player sooner if this weren't how we measured publication criteria (SOTA chasing). I believe this problem has only become worse. But this is how technology always advances, it isn't one technology getting better and better, but we see the composition of different technologies. Almost always where the replacement starts out as significantly worse than the existing status quo. So I'd argue we're leaving a lot of good work on the table by doing this. Certainly we have enough people working in ML that we can adequately do both, which is certainly much more optimal. You need both, but problem is we just compare to as if benchmarks are the only component of the story here.