Live data from Hacker News

Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

arxiv.org

151–160 of 177 posts

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#151
post #143

Earlier quoted context omitted.

Are there any books you recommend for deep learning that are written for developers who don't use math every day? I suppose the goal would be to understand deep learning so that we know enough of what's going on but not to get stuck in math concepts that we probably don't know and won't use.

I am/was in this scenario. I'm sure there are other resources out there specifically aimed at developers, but a book I'm reading now is "Deep Learning From Scratch" by Seth Weidman. He takes a different approach, by explaining concepts in three distinct methods: a mathematical way, by using diagrams and by showing the code. I like this approach because it allows me to connect the math to the problem, whereas otherwis…

The last commit in the repo of "Deep Learning from Scratch" was 5 years ago. It is hopelessly outdated. The field is changing very fast.

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#152

Earlier quoted context omitted.

> "the scaling hypothesis" Oh boy, don't get me started.... I first off should say that by no means do I think any of these people (at least those publishing) are dumb. You can also be a genius in one direction and a fucking idiot in another, and that's okay. Certainly describes me haha (well less on the genius side and more on the functioning idiot side. So take everything I say with a grain of salt). Don't get me w…

ML researchers saying they need "category theory" sounds like a way to try to convince mathematicians that their work is cool. You absolutely do not need category theory. Math is just models? Lol!

The parent didn't say category theory is necessary to conducting ML research, just that it could be useful. This point isn't particularly controversial. If you're interested in this niche of the field, I find Tai-Danae Bradley's work to be pretty cool! She has a site: https://www.math3ma.com/

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#153
post #126

Earlier quoted context omitted.

Yes, I have a very good point in fact. But the above comment purposely chooses not to argue with it, because it's easier to ignore it entirely and argue something else.

The problem is you presented something as a fact while it’s just a guess. Some people guess it’s an improved gradient flow, others guess it’s a smoother loss surface, someone else guesses it’s a shortcut for early layer information to reach later layers, etc. We don’t actually know why resnets work so well.

The point of that comment doesn't have anything to do with how ResNets actually work. You missed the actual point.

> We don’t actually know why resnets work so well.

Yes actually we do. We know, from the literature, that very deep neural networks suffered from vanishing gradients in their early layers in the same way traditional RNNs did. We know that was the motivation for introducing skip connections which gives us a hypothesis we can test. We can measure, using the test I described, the differences in the size of gradients in the early layers with and without skip connections. We can do this across many different problems for additional statistical power. We can analyze the linear case and see that the repeated matmults should lead to small gradients if their singular values are small. To ignore all of this and say that well we don't have a general proof that satisfies a mathematician so i guess we just don't know is silly.

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#154

Earlier quoted context omitted.

What percent of LLM knowledge requires proficiency in anything you mentioned? From what I’ve seen it’s a small percentage, and there’s no reason for most people to be put off by it. Everyone come on in the water is fine.

Between 0% and idk 70%? depending on what you're doing.

Looking at the theory as a whole it’s a very small minority.

I’m trying to think if it’s 0 percent outside of backprop…

Arguably high school math gets you quite a bit of understanding. After that in descending order I’d guess Linear Algebra, Statistics/Probability, Basic Calculus, Partial Derivatives…

In other words it’s not all or nothing. The easiest stuff gets you a lot of bang for your buck.

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#155

for those who want some maths-heavy stuff for deep learning, check francois fluret's book https://fleuret.org/francois/lbdl.html . the pdf is free but the print is so cute.

Has anyone figured out a way to print the Fleuret book on A4 paper? Every other page ends up upside down when I've tried it, which is problematic with a duplexer.

You can probably use pdftk to rotate every other page somehow.

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#156

Earlier quoted context omitted.

if you have no mathematical background at all this isn’t the book for you i think. that is not really advanced mathematics although a little notationally dense. there are many good materials such as the fantastic fast.ai course that don’t require such mathematical background. if you are motivated to learn about ML, then studying the topic can gradually be a route in to more mathematical knowledge so that equations li…

all I am saying is every deep learning book I have ever opened is filled with mathematical stuff like this. I want to learn the mathematics for it but I need a starting point. Isnt there atleast one book in the entire world written with this in mind?

This is because the authors aren’t trying to teach you anything. They are trying to show how smart they are and make a name for themselves with their peers. They could care less whether you learn. Almost no one doing deep learning will learn much from this.

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#157

It's hard to call comprehensive. Transformers - one page. A picture would be nice. No "prompt engineering", no "double deep". In fact words "prompt" and "double" aren't used at all. "Recognition" is used only once outside of bibliography just for reference. Looks like theory will not catch up with practice any time soon. With looming singularity it's bit worrying.

It's an introductory book though? I don't think it aims at being comprehensive

That said, i do agree that more on transformers would be nice since they're becoming quite central in every field of machine learning.

Prompt engineering is extremely new, vastly empirical, and theory on is still only beginning (though i do remember seeing some nice papers passing). It would probably be a mistake to include it in an introductory book

I have never heard of "double deep", what is that?

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#158

Earlier quoted context omitted.

It works better than you think, as long as you use GPT 4. See my answer to the other person ( https://news.ycombinator.com/item?id=38837646 ). A lot of negativity comes from people who goofed around with 3.X for a while, came away unimpressed, muttered something under their breath about stochastic parrots or Markov chains that sounded profound (at least to them), and never bothered to look any further. 4 is different…

That answer isn't very compelling as it is one of the most well known equations in ML. There are some very minor errors but nothing that changes the overall meaning. But you even seem to agree with me in your followup: don't rely on it, but use it. I'm only slightly stronger than you. And stop all this 3.5 vs 4 nonesense. We all know 4 is much better. But there's plenty of literature that shows its limits, especially…

You also don't understand stochastic parrots

You have a parrot that can paint original pictures, compose original songs and essays, and translate math into both English and program code?

I would like to buy your parrot. I'll keep it in my Chinese room. There used to be a guy in there, but he ran away screaming something about a basilisk.

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#159
Do people really finish these books cover to cover?

I have been working on Bishop's PRML, and it is extremely time consuming to really finish the book, and do all the exercises.

I saw a blog from a guy who did the same, and it took him 1500+ hours.

Not a single person in my masters program finished any of these books. They just did the courses and googled whatever else was necessary.

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#160
post #126

Earlier quoted context omitted.

The problem is you presented something as a fact while it’s just a guess. Some people guess it’s an improved gradient flow, others guess it’s a smoother loss surface, someone else guesses it’s a shortcut for early layer information to reach later layers, etc. We don’t actually know why resnets work so well.

The point of that comment doesn't have anything to do with how ResNets actually work. You missed the actual point. > We don’t actually know why resnets work so well. Yes actually we do. We know, from the literature, that very deep neural networks suffered from vanishing gradients in their early layers in the same way traditional RNNs did. We know that was the motivation for introducing skip connections which gives us…

You're doing it again - presenting guesses as facts. Why would a resnet - a batch normalized network using ReLU activations suffer from vanishing gradient problem? Does it? Have you actually done the experiment you've described? I have, and I didn't see gradients vanish. Sometimes gradients exploded - likely from a bad weights initialization (to be clear - that's a guess), and sometimes they didn't, but even when they didn't the networks never converged. The best we can do is to say: "skip connections seem to help training deep networks, and we have a few guesses as why, none of which is very convincing".

We know, from the literature

Let's look at the literature:

1. Training Very Deep Neural Networks: Rethinking the Role of Skip Connections: https://orbilu.uni.lu/bitstream/10993/47494/1/OyedotunAl%20I... they're making a hypothesis that skip connections might help prevent transformation of activations into singular matrices, which in turn could lead to unstable gradients (or not, it's a guess).

2. Improving the Trainability of Deep Neural Networks through Layerwise Batch-Entropy Regularization: https://openreview.net/pdf?id=LJohl5DnZf they are making some hypothesis about an optimal information flow through the network, and that a particular form of regularization helps improve this flow (no skip connections are needed).

3. Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifiers https://arxiv.org/abs/2203.08120: focus on initial conditions and propose better activation functions.

Clearly the issues are a bit more complicated than the vanishing gradients problem, and each of these papers offer a different explanation of why skip connections help.

It's similar to people building a bridge in 15th century - there was empirical evidence and intuition of how bridges should be built, but very little theory explaining that that evidence or intuition. Your statements are like "next time we should make the support columns thicker so that the bridge doesn't collapse", when in reality it collapsed due to the resonant oscillations induced by people marching on it in unison. Thicker columns will probably help, but they do nothing to improve understanding of the issue. They are just a guess.

That's why we need mathematicians looking at it, and attempting to formalize at least parts of the empirical evidence, so that someone, some day, will develop a compelling theory.

Post reply on HN