Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

arxiv.org

Post reply on HN
Print view View original

71–80 of 177 posts

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#71

Post by outrun86 »

angra_mainyu wrote: post #52
outrun86 wrote: post #49

Earlier quoted context omitted.

I’m just wrapping up a PhD in ML. The notation here is unnecessarily complex IMO. Notation can make things easier, or it can make things more difficult, depending on a number of factors.

Really? Coming from physics (B.Sc only) the notation is refreshingly familiar and straightforward. My topology and analysis classes were basically like this. In fact, this pdf is literally the resource I've been searching for as many others are far too ambiguous and handwavey focusing more on libraries and APIs than what's going on behind the scenes. If only there were a similar one for microeconomics and macroeconom…

Bishop’s Pattern Recognition and Machine Learning is one example that has tremendous depth and much clearer notation. Deep Learning by Goodfellow et al. is another example, albeit with less depth than Bishop.

I’m glad you’re enjoying the book. The approach is ideal for a very small subset of the ML population, no doubt that was their intention. I’m just weighing in that it’s entirely possible to cover this material with rigour yet much simpler notation. Even as someone who could parse this I’d go with other options.

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#72

Post by psyklic »

HighFreqAsuka wrote: post #51
p1esk wrote: post #50

Earlier quoted context omitted.

the reason skip connections improve performance is that they improve gradient flow in very deep networks. Can you prove this statement?

Empirically yes, I can consider a very deep fully-connected network, measure the gradients in each layer with and without skip connections, and compare. I can do this across multiple seeds and run a statistical test on the deltas.

Empirical studies are only useful until the system is mathematically understood. For example, I can construct transformer circuits where the skip connection (provably) purely adds noise.

I can also prove in particular cases the MLP's sole purpose is to remove the noise added from the skip connection.

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#73

Post by t_mann »

angra_mainyu wrote: post #52
outrun86 wrote: post #49

Earlier quoted context omitted.

I’m just wrapping up a PhD in ML. The notation here is unnecessarily complex IMO. Notation can make things easier, or it can make things more difficult, depending on a number of factors.

Really? Coming from physics (B.Sc only) the notation is refreshingly familiar and straightforward. My topology and analysis classes were basically like this. In fact, this pdf is literally the resource I've been searching for as many others are far too ambiguous and handwavey focusing more on libraries and APIs than what's going on behind the scenes. If only there were a similar one for microeconomics and macroeconom…

Re your question on economics books, I think Advanced Macroeconomics by David Romer could fit your bill. It goes a lot into why the math is the way it is (arguably more interesting, like another poster said). Modern macroeconomics is also built on microeconomics, and to that extent it's covered in the book, so you're sort of getting two-for-one here.

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#74

Post by c7b »

p1esk wrote: post #50
HighFreqAsuka wrote: post #45

Earlier quoted context omitted.

Just read the section on ResNets (Section 1.5) and tell me if you think that's the best way to explain ResNets to literally anyone. Tell me if, from that description, you take away that the reason skip connections improve performance is that they improve gradient flow in very deep networks.

the reason skip connections improve performance is that they improve gradient flow in very deep networks. Can you prove this statement?

Neither do the authors in the book, and I'd argue that after (only) reading the book, the reader wouldn't be equipped to attempt this either (see my other post in this thread), so I think the parent poster has a point.

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#75

Post by ivancho »

HybridCurve wrote: post #25

As someone who has a deeper knowledge of programming rather than math, I find the mathematical notation here to be harder to understand than the code (even in a programming language I do not know). Does anyone with a stronger mathematical background here find it easier to understand the math as written more easily than the source code?

I have a strong mathematical background, and I found the notation completely insane. Right out of the gate in chapter 1 we get a definition that has subscript indices in the subscript index and a summation with subscripts in the superscript, and then composed in a giant function chain. Later we get to 4-level subscripts deep, invent at least 3 new infix operators, define 30 new symbols from 3 different alphabets and we're barely at page 100 out of 600. I have no idea who is supposed to follow and digest this

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#76

Post by tnecniv »

layer8 wrote: post #31
HybridCurve wrote: post #25

As someone who has a deeper knowledge of programming rather than math, I find the mathematical notation here to be harder to understand than the code (even in a programming language I do not know). Does anyone with a stronger mathematical background here find it easier to understand the math as written more easily than the source code?

Mathematical notation is more concise, which may take some getting used to. One reason is that it is optimized for handwriting. Handwriting program code would be very tedious, so you can see why mathematical notation is the way it is. Apart from that, there is no “the code” equivalent. Mathematical notation is for stating mathematical facts or propositions. That’s different from the purpose of the code you would writ…

The last part was a big hurdle for me as an early undergrad. I was a fairly strong programmer toward the end of high school, and was trying to think of math as programming. That worked for the fairly algorithmic high school stuff and I got good grades, but it made I was awful at writing proofs. I also went through a phase where I used all the logical notation and rules to manipulate it possible in order to make proofs more algorithmic to me, but that both didn’t work well for me and produced some downright unreadable results.

Mathematical notation is really a shorthand for words, like you’d read text. The equals sign is literally short for equals. The added benefit, as others have pointed out, is that a good notation can sometimes be clearer than words because it makes certain conclusions almost obvious. You’ve done the hard part in finding a notation that captures exactly the idea to be demonstrated in its encoding, and the result is a very clean manipulation of your notation.

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#77

Post by Koshkin »

layer8 wrote: post #31
HybridCurve wrote: post #25

As someone who has a deeper knowledge of programming rather than math, I find the mathematical notation here to be harder to understand than the code (even in a programming language I do not know). Does anyone with a stronger mathematical background here find it easier to understand the math as written more easily than the source code?

Mathematical notation is more concise, which may take some getting used to. One reason is that it is optimized for handwriting. Handwriting program code would be very tedious, so you can see why mathematical notation is the way it is. Apart from that, there is no “the code” equivalent. Mathematical notation is for stating mathematical facts or propositions. That’s different from the purpose of the code you would writ…

> Mathematical notation is for stating mathematical facts or propositions.

And as such it is way too often abused. Because the (original, and the most useful) purpose of mathematical notation is to enable calculation, i.e., in a general sense, to make it possible to obtain results by manipulating symbols according certain rules.

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#78

Post by layer8 »

Koshkin wrote: post #77
layer8 wrote: post #31

Earlier quoted context omitted.

Mathematical notation is more concise, which may take some getting used to. One reason is that it is optimized for handwriting. Handwriting program code would be very tedious, so you can see why mathematical notation is the way it is. Apart from that, there is no “the code” equivalent. Mathematical notation is for stating mathematical facts or propositions. That’s different from the purpose of the code you would writ…

> Mathematical notation is for stating mathematical facts or propositions. And as such it is way too often abused. Because the (original, and the most useful) purpose of mathematical notation is to enable calculation , i.e., in a general sense, to make it possible to obtain results by manipulating symbols according certain rules.

I see the steps of a calculation as stating a sequence of mathematical facts, so that’s just an instance of the general definition.

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#80

Post by Koshkin »

layer8 wrote: post #78
Koshkin wrote: post #77

Earlier quoted context omitted.

> Mathematical notation is for stating mathematical facts or propositions. And as such it is way too often abused. Because the (original, and the most useful) purpose of mathematical notation is to enable calculation , i.e., in a general sense, to make it possible to obtain results by manipulating symbols according certain rules.

I see the steps of a calculation as stating a sequence of mathematical facts, so that’s just an instance of the general definition.

Sure, but the whole point is to avoid the need to do that! Manipulating symbols is the way to automate reasoning, i.e. to get to a result while completely ignoring said "facts." Using the symbols to merely "state the facts" is abuse (of the reader, mostly).
Post reply on HN