Matrix Calculus (For Machine Learning and Beyond)

edit: On the math side I've encountered one that seemed unique, as I haven't seen anything like this elsewhere: https://irregular-rhomboid.github.io/2022/12/07/applied-math.... However, this only points out courses that he enrolled in his math education that he thinks is relevant to ML, each course is given a very short description and or motivation as to the usefulness it has to ML.

I like this concluding remarks:

Through my curriculum, I learned about a broad variety of subjects that provide useful ideas and intuitions when applied to ML. Arguably the most valuable thing I got out of it is a rough map of mathematics that I can use to navigate and learn more advanced topics on my own.

Having already been exposed to these ideas, I wasn’t confused when I encountered them in ML papers. Rather, I could leverage them to get intuition about the ML part.

Strictly speaking, the only math that is actually needed for ML is real analysis, linear algebra, probability and optimization. And even there, your mileage may vary. Everything else is helpful, because it provides additional language and intuition. But if you’re trying to tackle hard problems like alignment or actually getting a grasp on what large neural nets actually do, you need all the intuition you can get. If you’re already confused about the simple cases, you have no hope of deconfusing the complex ones.

So in this particular instance since we are talking about matrix calculus it’s a type of multivariable calculus where you’re dealing with functions which take matrices as inputs and “matrix-valued functions” (ie functions which return matrices as an output).

Calculus is used for a lot of things but for example if you have a continuous function you can use calculus to find maxima and minima, inflection points etc. Since the focus here is machine learning, one of the most important things you want to be able to do is gradient descent to optimise some cost function by tweaking the weights on your model. The gradient here is a vector field which at every point in the space of your model points in the direction of steepest ascent[1]. So if you want to go down you take the gradient at a particular point and go exactly in the opposite direction. That means you know exactly what model weight tweaks will cause the biggest decrease in your cost function the next time you do a training run.

[1] To imagine the gradient, think of an old-school contour map like you’re going to do a hike or something. This is a “scalar field” (a map from spatial coordinates to a scalar value - altitude). The contour lines on the map link points of equal altitude. The gradient is a “vector field” which is a map from spatial coordinates to a vector. So imagine at every point on your contour map there was a little arrow that pointed in the direction of steepest ascent. Because this is matrix calculus you will be dealing with “matrix fields” (maps from spatial coordinates to matrices) as well. So for example say you did a measurement of stress in a steel beam. At each point you would have a “stress tensor” which says what the forces are at that point and in which direction they are pointing. This is a “tensor field” (map from spatial coordinates to a tensor) and a tensor is like a multidimensional matrix but with some additional rules about how it transforms.

I wouldn't teach it this way, but I would definitely take the taylor expansion and define the grad vector as the one to make the best local linear approximation. This tells you that the grad lives in the same space, i.e. same dimensions.

Of course, you can always switch things around if you want to calculate things and put in transposes where they belong. But I find it insane to take the unnatural view point as standard, which I find a lot of papers do.

I'm a simple-minded physicist. I just know if you apply the same coordinate transformation to the gradient and to the displacement vector, you get the wrong answer.

My usual reference is Schutz's Geometrical Methods of Mathematical Physics, and he defines the gradient as df, but other sources call that the "differential" and say the gradient is what you get if you use the metric to raise the indices of df.

But that raised-index gradient (i.e. g(df)), is weird and non-physical. It doesn't behave properly under coordinate transformations. So I'm not sure why folks use that definition.

You can see difference by looking at the differential in polar coordinates. If you have f=x+y, then df=dx+dy=(cos th + sin th)dr + r(cos th - sin th)d th. If you pretend this is instead a vector and transform it, you'd get "df"=(cos th + sin th)dr + (1/r)(cos th - sin th)d th, which just gives the wrong answer.

To be specific, if v=(1,1) in cartesian (ex,ey), then df(v)=2. But (1,1) in cartesian is (1,1/r) in polar (er, etheta). The "proper" df still gives 2, but the "weird metric one" gives 1+1/r^2, since you get the 1/r factor twice, instead of a 1/r and a balancing r.