Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Matrix Calculus for Deep Learning

parrt.cs.usfca.edu

Post reply on HN
Print view View original

21–30 of 84 posts

Re: Matrix Calculus for Deep Learning

#21

Post by jacobolus »

posterboy wrote: post #20
jacobolus wrote: post #8

Earlier quoted context omitted.

Typographic advice: the body text has very long lines in a desktop browser, which makes it a bit slow and tiring to read. I’d say the ideal is somewhere between 1/2 and 2/3 this length. I’d recommend keeping the same width on screen but bumping the font size up by 30%. As an extra minor nit, italicizing functions like sin , etc. is also somewhat unconventional in mathematical typesetting.

it's a variable name for a polymorphic function, taylor series, euler formular, etc. depending on context. This is a significant difference to singular types, but being a variable name to an abstract concept is in principle no different to typesetting x . This goes neatly with "everything is an objectreference" and might be more of a programmer's perspective.

In mathematical notation it gets confusing because several italic letters in a row are otherwise interpreted as separate variables.

Re: Matrix Calculus for Deep Learning

#22

Post by lprd »

In school, I didn't make it much past basic calculus/algebra. As a self-taught programmer (my highest level of education is a high-school diploma), I seriously wish I could go back and put more effort into math. I love looking at these types of topics, but I have absolutely no clue what I'm looking at.

If anyone can recommend any books, courses, or any other material that starts from high-school level math, and gradually increases in complexity, I would love to look at it.

Cheers :)

Re: Matrix Calculus for Deep Learning

#23

Post by j2kun »

lprd wrote: post #22

In school, I didn't make it much past basic calculus/algebra. As a self-taught programmer (my highest level of education is a high-school diploma), I seriously wish I could go back and put more effort into math. I love looking at these types of topics, but I have absolutely no clue what I'm looking at. If anyone can recommend any books, courses, or any other material that starts from high-school level math, and gradu…

I am writing a book that I hope can serve such a purpose. Would you be interested in taking a look? If so, shoot me an email at mathintersectprogramming@gmail.com

Fair warning, I have shown it to a programmer who claimed some level of "math phobia," and they said the first chapter was too difficult. I rewrote that chapter since, and I think it is better, but I could use some feedback :)

Re: Matrix Calculus for Deep Learning

#24

Post by sydl »

Thanks for this great contribution.

I would like to be able to read the math in DL papers. (sorry I'm asking for something that it's too broad)

1) How much does this document cover the notations in those papers. 2) When I read a paper and if I am not sure what the math means, does that mean that I did not grok the subject yet, or the math presented in that paper goes beyond the math given in this Matrix Calculus document (assuming I studied well this document).

Re: Matrix Calculus for Deep Learning

#25

Post by pacbard »

sydl wrote: post #24

Thanks for this great contribution. I would like to be able to read the math in DL papers. (sorry I'm asking for something that it's too broad) 1) How much does this document cover the notations in those papers. 2) When I read a paper and if I am not sure what the math means, does that mean that I did not grok the subject yet, or the math presented in that paper goes beyond the math given in this Matrix Calculus docu…

I would say try to get as much of the intuition behind the math as you can. Knowing what an equation means rather than fully understanding the Greek notation is what it matters for practice. If you want to substantially contribute to the theoretical CS literature, you will need to have a good handle of the notation (for obvious reasons).

Note: I come from math and Econ, so the split between practitioner and theorists might be different for CS/ML.

Re: Matrix Calculus for Deep Learning

#26

Post by zwieback »

Thanks so much for this. I have no interest in deep learning (at the moment) but I was working through some papers about the Lucas Kanade tracker and this paper explains some of the underlying math in just the right amount of detail. The authors usually show the beginning and end point and just say something like "using the chain rule" we arrive at ... It took me a while to understand what they were saying and this paper helps a lot.

The math is super easy but keeping all the notation s and conventions in my head is hard, I've never seen it laid out this nicely before. Thanks!

Re: Matrix Calculus for Deep Learning

#27

Post by mlevental »

jules wrote: post #5
jph00 wrote: post #4

Jeremy here. Here to answer any questions or comments that you have. But more importantly - I need to mention that Terence Parr did nearly all the work on this. He shared my passion for making something that anyone could read on any device to such an extent that he ended up creating a new tool for generating fast, mobile-friendly math-heavy texts: https://github.com/parrt/bookish . (We tried Katex, Mathjax, and prett…

What do you think about the index notation physicists use for tensor calculus?

index notation isn't useful for calculation it's useful for algebra i.e. if you want closed form solution to tensor equations. my distinct impression is that in ML no one cares about that because eventually everything gets a numerical treatment.

Re: Matrix Calculus for Deep Learning

#28

Post by adyavanapalli »

Matrix calculus is a bit screwy when you realize that there are two possible notations to represent matrix derivatives (numerator vs. denominator layout; numerator layout is used in this guide). Plus, the notation is not very "speaking" for doing calculations unless you commit to memory some basic results.. which is why, as a physicist, I would recommend working in tensor calculus notation during calculations, and translating back to matrix notation for writing the results.

Re: Matrix Calculus for Deep Learning

#29

Post by smith-kyle »

jph00 wrote: post #4

Jeremy here. Here to answer any questions or comments that you have. But more importantly - I need to mention that Terence Parr did nearly all the work on this. He shared my passion for making something that anyone could read on any device to such an extent that he ended up creating a new tool for generating fast, mobile-friendly math-heavy texts: https://github.com/parrt/bookish . (We tried Katex, Mathjax, and prett…

Hi Jeremy, thanks for creating so many great learning resources. Myself and a fellow ML enthusiast are starting a deep learning meetup here in Phoenix next month. Do you have any advice for creating a welcoming environment for people to learn/teach? Thanks!

Re: Matrix Calculus for Deep Learning

#30

Post by parrt »

zwieback wrote: post #26

Thanks so much for this. I have no interest in deep learning (at the moment) but I was working through some papers about the Lucas Kanade tracker and this paper explains some of the underlying math in just the right amount of detail. The authors usually show the beginning and end point and just say something like "using the chain rule" we arrive at ... It took me a while to understand what they were saying and this p…

Hiya. That's funny because it's exactly what caused us to write this article. Jeremy and I were working on an automatic differentiation tool and couldn't find any description of the appropriate matrix calculus that explained the steps. Everything is just providing the solution without the intervening steps. We decided to write it down so we never have to figure out the notation again. haha
Post reply on HN