Matrix Calculus for Deep Learning • Hacker News Board

Re: Matrix Calculus for Deep Learning

#31

Post by parrt » Tue, Jan 30, 2018, 8:58 PM UTC

adyavanapalli wrote: post #28

Matrix calculus is a bit screwy when you realize that there are two possible notations to represent matrix derivatives (numerator vs. denominator layout; numerator layout is used in this guide). Plus, the notation is not very "speaking" for doing calculations unless you commit to memory some basic results.. which is why, as a physicist, I would recommend working in tensor calculus notation during calculations, and tr…

I was also surprised when I saw that there was no standard notation for Jacobian matrices. We use the numerator notation in the article, but point out that there are papers that use the denominator notation. I think I remember from engineering school that we used numerator notation so we stuck with that.

Re: Matrix Calculus for Deep Learning

#32

Post by kaffeinecoma » Tue, Jan 30, 2018, 9:08 PM UTC

Wow, this is really a great resource. I wish it had been available a few years back when I took the free online version of CS231n. The hardest part (for me, anyway) was the long-forgotten Calculus needed for backprop. Especially as applied to matrices. I struggled at the time to find accessible explanations of many of the matrix operations, and you seem to have it all laid out here. Thank you.

Re: Matrix Calculus for Deep Learning

#33

Post by calebh » Tue, Jan 30, 2018, 9:08 PM UTC

Fortunately there is a website now capable of doing matrix calculus! http://www.matrixcalculus.org

Mathematica doesn't seem to be able to do matrix calculus, which surprised me quite a bit.

Re: Matrix Calculus for Deep Learning

#34

Post by parrt » Tue, Jan 30, 2018, 9:11 PM UTC

calebh wrote: post #33

Fortunately there is a website now capable of doing matrix calculus! http://www.matrixcalculus.org Mathematica doesn't seem to be able to do matrix calculus, which surprised me quite a bit.

Wow! Great little calculator. Thanks for pointing us at it.

Re: Matrix Calculus for Deep Learning

#35

Post by BucketSort » Tue, Jan 30, 2018, 9:14 PM UTC

calebh wrote: post #33

Fortunately there is a website now capable of doing matrix calculus! http://www.matrixcalculus.org Mathematica doesn't seem to be able to do matrix calculus, which surprised me quite a bit.

Indeed, most tools surprisingly lack this ability. I was shocked when I needed to break my calculations down to a piece-wise form when doing matrix calc with SymPy.

Re: Matrix Calculus for Deep Learning

#36

Post by adyavanapalli » Tue, Jan 30, 2018, 9:15 PM UTC

calebh wrote: post #33

Fortunately there is a website now capable of doing matrix calculus! http://www.matrixcalculus.org Mathematica doesn't seem to be able to do matrix calculus, which surprised me quite a bit.

Note: This website presumes denominator layout, which is different from what is used in the guide.

Re: Matrix Calculus for Deep Learning

#37

Post by singularity2001 » Tue, Jan 30, 2018, 9:23 PM UTC

marrowgari wrote: post #3

If you're looking at this with the intention of getting started in Deep Learning and feeling overwhelmed by the math then Andrew Ng offers a great course on Coursera that goes over all of the formulas needed to calculate the forward propagation, loss computation, backward propagation, and gradient descent. Highly recommend it for anyone interested in breaking into the field of machine learning.

if you want to get started with machine learning you should just start with Keras and do some theory later.

Re: Matrix Calculus for Deep Learning

#38

Post by jph00 » Tue, Jan 30, 2018, 9:25 PM UTC

lprd wrote: post #22

In school, I didn't make it much past basic calculus/algebra. As a self-taught programmer (my highest level of education is a high-school diploma), I seriously wish I could go back and put more effort into math. I love looking at these types of topics, but I have absolutely no clue what I'm looking at. If anyone can recommend any books, courses, or any other material that starts from high-school level math, and gradu…

Well... this paper is really designed to be accessible with just high school math, if you take your time (a few weeks or months) and follow the references. Any time it relies on some concept, it includes a link to learning more about that concept, and also has a link to a forum where you can ask questions if you get stuck. There's also a table of all notation used.

If you give it a go and find you're not successful, I'd be interested to hear where is the first point where you got stuck and couldn't get unstuck, since that would suggest a need for us to improve our paper!

Re: Matrix Calculus for Deep Learning

#39

Post by edhu2017 » Tue, Jan 30, 2018, 9:44 PM UTC

jph00 wrote: post #4

Jeremy here. Here to answer any questions or comments that you have. But more importantly - I need to mention that Terence Parr did nearly all the work on this. He shared my passion for making something that anyone could read on any device to such an extent that he ended up creating a new tool for generating fast, mobile-friendly math-heavy texts: https://github.com/parrt/bookish . (We tried Katex, Mathjax, and prett…

In the article, you say the gradient of 2x + y^8 is [1, 7y^8]. Shouldn't it be [2, 7y^8]?

Re: Matrix Calculus for Deep Learning

#40

Post by smrtinsert » Tue, Jan 30, 2018, 9:47 PM UTC

Thanks for this. Was taking Andrew Ng's course but the way he glosses over the calculus and then expects the student to understand the implications at end of lecture was a turn off so I dropped it. I hated the feeling I wasn't learning, just memorizing solutions.