Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Matrix Calculus for Deep Learning

parrt.cs.usfca.edu

Post reply on HN
Print view View original

61–70 of 84 posts

Re: Matrix Calculus for Deep Learning

#61

Post by improbable22 »

parrt wrote: post #31
adyavanapalli wrote: post #28

Matrix calculus is a bit screwy when you realize that there are two possible notations to represent matrix derivatives (numerator vs. denominator layout; numerator layout is used in this guide). Plus, the notation is not very "speaking" for doing calculations unless you commit to memory some basic results.. which is why, as a physicist, I would recommend working in tensor calculus notation during calculations, and tr…

I was also surprised when I saw that there was no standard notation for Jacobian matrices. We use the numerator notation in the article, but point out that there are papers that use the denominator notation. I think I remember from engineering school that we used numerator notation so we stuck with that.

This is what index notation is good for, and I encourage everyone to learn it. Jacobians are dx_a/dx_b, two indices, and clearly b belongs to the derivative. Whether it's rows or columns is an implementation detail of how you're storing these numbers.

Index notation also seems natural for programming: an element A[i,j] or a slice Z[3,4,:] are precisely this.

Re: Matrix Calculus for Deep Learning

#62

Post by mewert »

adyavanapalli wrote: post #28

Matrix calculus is a bit screwy when you realize that there are two possible notations to represent matrix derivatives (numerator vs. denominator layout; numerator layout is used in this guide). Plus, the notation is not very "speaking" for doing calculations unless you commit to memory some basic results.. which is why, as a physicist, I would recommend working in tensor calculus notation during calculations, and tr…

I agree. If your Matrix calculus involves more complicated use of derivative operators you can't treat it like linear algebra anymore. Better to break it down into something like tensor notation first and back to matrices at the end. https://en.wikipedia.org/wiki/Del. Specifically I was always confused by the material derivative of a vector field when presented as either Matrix or vector calculus. If you represent it in tensor notation ( or explicitly break it out as operations on basis vectors ) it works out nicely.

Re: Matrix Calculus for Deep Learning

#64

Post by improbable22 »

calebh wrote: post #56
improbable22 wrote: post #51

Earlier quoted context omitted.

Like this, perhaps? A = {{1,2},{3,4}} vec = {x^2, x^3} D[vec.A.vec, x] Or perhaps like this, again the table of derivatives: xvec = {x1,x2} Table[ D[xvec.A.xvec,x] ,{x,xvec}] (all untested... one typo caught...)

Mathematica can definitely compute the derivatives if you fix the size of the matrix. This isn't very useful if you're trying to compute the derivative of an expression with arbitrary sized matrices.

You can do general matrices too, what do you have in mind?

aa[x_] = {{1, 2}, {3, 4}} x

bb[x_] = {x^2, x^3}

D[ a[x].b[x] , x, x] (* for any suitable tensors *)

% /. {a -> aa, b -> bb}

Re: Matrix Calculus for Deep Learning

#65

Post by dbetteridge »

parrt wrote: post #12
jacobolus wrote: post #8

Earlier quoted context omitted.

Typographic advice: the body text has very long lines in a desktop browser, which makes it a bit slow and tiring to read. I’d say the ideal is somewhere between 1/2 and 2/3 this length. I’d recommend keeping the same width on screen but bumping the font size up by 30%. As an extra minor nit, italicizing functions like sin , etc. is also somewhat unconventional in mathematical typesetting.

I agree that the font should be bigger. I need to learn more CSS in order to switch between font sizes per platform. The font of the text is easy but all of the images were generated from latex using a specific font size. I need to scale the in-line equation images as the font size bumps up.

the magic incantation here is probably media queries

@media (max-width: 768px) {

    p {

     font-size: 1rem;
    }
}

Re: Matrix Calculus for Deep Learning

#66

Post by jph00 »

jules wrote: post #59
jph00 wrote: post #11

Earlier quoted context omitted.

I don't find it very accessible, myself - but I'm not a physicist, so the materials using or about that notation aren't aimed at me. The only tensor notation I've been happy with is that used in J ( http://www.jsoftware.com ), which is simple, flexible, and concise. There's also some nice-enough modern notation used in this excellent review: http://www.cs.cmu.edu/~christos/courses/826-resources/PAPERS...

The notation is very simple: you write tensor expressions with indices, and repeated indices are implicitly summed over. For example, matrix multiply: A_ij = B_ik C_kj Differentiating this with respect to variable l: ∂_l A_ij = ∂_l (B_ik C_kj) = (∂_l B_ik) C_kj + B_ik (∂_l C_kj) By writing out indices you can just use the rules for scalar derivatives.

Oh in that case I'm just confused - that's what I know as Einstein notation. Modern physics papers seemed to use much more complex notation, but I probably just misunderstood.

If we're talking about Einstein notation, then I'm a fan - `np.einsum()` is often a great way to create fast tensor computations with minimal code.

Re: Matrix Calculus for Deep Learning

#67

Post by jph00 »

calebh wrote: post #58
adyavanapalli wrote: post #36

Earlier quoted context omitted.

Note: This website presumes denominator layout, which is different from what is used in the guide.

Does the layout matter as long as you're consistent? Do the deep learning libraries presume that you're using a certain layout?

The deep learning libraries largely hide all the calculus - it's all automated.

Re: Matrix Calculus for Deep Learning

#68

Post by jph00 »

blt wrote: post #53
sydl wrote: post #24

Thanks for this great contribution. I would like to be able to read the math in DL papers. (sorry I'm asking for something that it's too broad) 1) How much does this document cover the notations in those papers. 2) When I read a paper and if I am not sure what the math means, does that mean that I did not grok the subject yet, or the math presented in that paper goes beyond the math given in this Matrix Calculus docu…

While matrix derivatives are important, there is also a lot of other math in DL papers. In particular, a lot of the probability side concerns expectations, KL divergences, entropy, etc., which are all defined in terms of integrals or sums. You need undergraduate-level probability background.

The first 5 chapters of the Goodfellow deep learning book are a great resource for understanding the probability, linear algebra, optimization, and information theory you need to digest deep learning papers.

Re: Matrix Calculus for Deep Learning

#69

Post by joshgel »

jph00 wrote: post #42
smrtinsert wrote: post #40

Thanks for this. Was taking Andrew Ng's course but the way he glosses over the calculus and then expects the student to understand the implications at end of lecture was a turn off so I dropped it. I hated the feeling I wasn't learning, just memorizing solutions.

You might prefer the approach at http://course.fast.ai - all the concepts are taught with code, instead of math, and understanding is developed by running experiments.

I did both and found fast.ai so much easier to understand for someone without a background in math, like me.

Re: Matrix Calculus for Deep Learning

#70

Post by hinkley »

So are we turning machine learning into a Euclidean distance calculation [with] many dimensions, with different weights for each dimension?

That’s... not that sexy. But at least it makes sense to anyone with an undergrad degree in CS or math, which is something neural networks never accomplished.

Post reply on HN