Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Matrix Calculus for Deep Learning

parrt.cs.usfca.edu

Post reply on HN
Print view View original

51–60 of 84 posts

Re: Matrix Calculus for Deep Learning

#51

Post by improbable22 »

SoerenL wrote: post #48
improbable22 wrote: post #45

Earlier quoted context omitted.

Mathematica can absolutely do all of this! D[ matrix ,x] ... or things like this: f[x_,y_] := x^2 + Sin[y] vars={x,y}; Table[ D[f[x,y],var1, var2], {var1,vars}, {var2,vars}] // MatrixForm

But how do you compute the derivative of x' A x in Mathematica (x being a vector and A being a matrix)? What you have pointed out is only scalar derivatives, if I am not mistaken here.

Like this, perhaps?

A = {{1,2},{3,4}}

vec = {x^2, x^3}

D[vec.A.vec, x]

Or perhaps like this, again the table of derivatives:

xvec = {x1,x2}

Table[ D[xvec.A.xvec,x] ,{x,xvec}]

(all untested... one typo caught...)

Re: Matrix Calculus for Deep Learning

#52

Post by claar »

improbable22 wrote: post #51
SoerenL wrote: post #48

Earlier quoted context omitted.

But how do you compute the derivative of x' A x in Mathematica (x being a vector and A being a matrix)? What you have pointed out is only scalar derivatives, if I am not mistaken here.

Like this, perhaps? A = {{1,2},{3,4}} vec = {x^2, x^3} D[vec.A.vec, x] Or perhaps like this, again the table of derivatives: xvec = {x1,x2} Table[ D[xvec.A.xvec,x] ,{x,xvec}] (all untested... one typo caught...)

http://www.wolframalpha.com/input/?i=D%5B%7Bx%5E2,+x%5E3%7D....

Re: Matrix Calculus for Deep Learning

#53

Post by blt »

sydl wrote: post #24

Thanks for this great contribution. I would like to be able to read the math in DL papers. (sorry I'm asking for something that it's too broad) 1) How much does this document cover the notations in those papers. 2) When I read a paper and if I am not sure what the math means, does that mean that I did not grok the subject yet, or the math presented in that paper goes beyond the math given in this Matrix Calculus docu…

While matrix derivatives are important, there is also a lot of other math in DL papers. In particular, a lot of the probability side concerns expectations, KL divergences, entropy, etc., which are all defined in terms of integrals or sums. You need undergraduate-level probability background.

Re: Matrix Calculus for Deep Learning

#54

Post by lprd »

jph00 wrote: post #38
lprd wrote: post #22

In school, I didn't make it much past basic calculus/algebra. As a self-taught programmer (my highest level of education is a high-school diploma), I seriously wish I could go back and put more effort into math. I love looking at these types of topics, but I have absolutely no clue what I'm looking at. If anyone can recommend any books, courses, or any other material that starts from high-school level math, and gradu…

Well... this paper is really designed to be accessible with just high school math, if you take your time (a few weeks or months) and follow the references. Any time it relies on some concept, it includes a link to learning more about that concept, and also has a link to a forum where you can ask questions if you get stuck. There's also a table of all notation used. If you give it a go and find you're not successful,…

Hi Jeremy,

I greatly appreciate your response! I will take a long look at this paper again and attempt to digest it.

Thanks again!

Re: Matrix Calculus for Deep Learning

#55

Post by lprd »

KSS42 wrote: post #50
lprd wrote: post #22

In school, I didn't make it much past basic calculus/algebra. As a self-taught programmer (my highest level of education is a high-school diploma), I seriously wish I could go back and put more effort into math. I love looking at these types of topics, but I have absolutely no clue what I'm looking at. If anyone can recommend any books, courses, or any other material that starts from high-school level math, and gradu…

I would start by watching the Linear Algebra and Calculus videos by 3Blue1Brown. This will give you an intuitive understanding of both. https://www.youtube.com/playlist?list=PLZHQObOWTQDPD3MizzM2x... https://www.youtube.com/playlist?list=PLZHQObOWTQDMsr9K-rj53...

Thank you!

Re: Matrix Calculus for Deep Learning

#56

Post by calebh »

improbable22 wrote: post #51
SoerenL wrote: post #48

Earlier quoted context omitted.

But how do you compute the derivative of x' A x in Mathematica (x being a vector and A being a matrix)? What you have pointed out is only scalar derivatives, if I am not mistaken here.

Like this, perhaps? A = {{1,2},{3,4}} vec = {x^2, x^3} D[vec.A.vec, x] Or perhaps like this, again the table of derivatives: xvec = {x1,x2} Table[ D[xvec.A.xvec,x] ,{x,xvec}] (all untested... one typo caught...)

Mathematica can definitely compute the derivatives if you fix the size of the matrix. This isn't very useful if you're trying to compute the derivative of an expression with arbitrary sized matrices.

Re: Matrix Calculus for Deep Learning

#57

Post by calebh »

parrt wrote: post #34
calebh wrote: post #33

Fortunately there is a website now capable of doing matrix calculus! http://www.matrixcalculus.org Mathematica doesn't seem to be able to do matrix calculus, which surprised me quite a bit.

Wow! Great little calculator. Thanks for pointing us at it.

You should add a link in the resources section

Re: Matrix Calculus for Deep Learning

#58

Post by calebh »

adyavanapalli wrote: post #36
calebh wrote: post #33

Fortunately there is a website now capable of doing matrix calculus! http://www.matrixcalculus.org Mathematica doesn't seem to be able to do matrix calculus, which surprised me quite a bit.

Note: This website presumes denominator layout, which is different from what is used in the guide.

Does the layout matter as long as you're consistent? Do the deep learning libraries presume that you're using a certain layout?

Re: Matrix Calculus for Deep Learning

#59

Post by jules »

jph00 wrote: post #11
jules wrote: post #5

Earlier quoted context omitted.

What do you think about the index notation physicists use for tensor calculus?

I don't find it very accessible, myself - but I'm not a physicist, so the materials using or about that notation aren't aimed at me. The only tensor notation I've been happy with is that used in J ( http://www.jsoftware.com ), which is simple, flexible, and concise. There's also some nice-enough modern notation used in this excellent review: http://www.cs.cmu.edu/~christos/courses/826-resources/PAPERS...

The notation is very simple: you write tensor expressions with indices, and repeated indices are implicitly summed over. For example, matrix multiply:

   A_ij = B_ik C_kj
Differentiating this with respect to variable l:

   ∂_l A_ij = ∂_l (B_ik C_kj) = (∂_l B_ik) C_kj + B_ik (∂_l C_kj)
By writing out indices you can just use the rules for scalar derivatives.

Re: Matrix Calculus for Deep Learning

#60

Post by archgoon »

jph00 wrote: post #4

Jeremy here. Here to answer any questions or comments that you have. But more importantly - I need to mention that Terence Parr did nearly all the work on this. He shared my passion for making something that anyone could read on any device to such an extent that he ended up creating a new tool for generating fast, mobile-friendly math-heavy texts: https://github.com/parrt/bookish . (We tried Katex, Mathjax, and prett…

Question; when I learned Vector Calculus back in college, I used Marsden & Tromba as our text book where they equate the derivative of a function from R^n->R^m with the Jacobian. Is matrix calculus the same thing, just a slightly different notation?
Post reply on HN