Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Matrix Calculus (For Machine Learning and Beyond)

arxiv.org

Post reply on HN
Print view View original

11–20 of 31 posts

Re: Matrix Calculus (For Machine Learning and Beyond)

#11

Post by C-x_C-f »

sfpotter wrote: post #8

If you want to get handy with matrix calculus, the real prerequisite is being comfortable with Taylor expansions and linear algebra. In a graduate numerical optimization class I took over a decade ago, the professor spent 10 minutes on the first day deriving some matrix calculus identity by working out the expressions for partial derivatives using simple calculus rules and a lot of manual labor. Then, as the class wa…

> Also, don't forget the Jacobian and gradient aren't the same thing!

Every gradient is a Jacobian but not every Jacobian is a gradient.

If you have a map f from R^n to R^m then the Jacobian at a point x is an m x n matrix which linearly approximates f at x. If m = 1 (namely if f is a scalar function) then the Jacobian is exactly the gradient.

If you already know about gradients (e.g. from physics or ML) and can't quite wrap your head around the Jacobian, the following might help (it's how I first got to understand Jacobians better):

1. write your function f from R^n to R^m as m scalar functions f_1, ..., f_m, namely f(x) = (f_1(x), ..., f_m(x))

2. take the gradient of f_i for each i

3. make an m x n matrix where the i-th row is the gradient of f_i

The matrix you build in step 3 is precisely the Jacobian. This is obvious if you know the definition and it's not a mathematically remarkable fact but for me at least it was useful to demystify the whole thing.

Re: Matrix Calculus (For Machine Learning and Beyond)

#12

Post by sfpotter »

C-x_C-f wrote: post #11
sfpotter wrote: post #8

If you want to get handy with matrix calculus, the real prerequisite is being comfortable with Taylor expansions and linear algebra. In a graduate numerical optimization class I took over a decade ago, the professor spent 10 minutes on the first day deriving some matrix calculus identity by working out the expressions for partial derivatives using simple calculus rules and a lot of manual labor. Then, as the class wa…

> Also, don't forget the Jacobian and gradient aren't the same thing! Every gradient is a Jacobian but not every Jacobian is a gradient. If you have a map f from R^n to R^m then the Jacobian at a point x is an m x n matrix which linearly approximates f at x. If m = 1 (namely if f is a scalar function) then the Jacobian is exactly the gradient. If you already know about gradients (e.g. from physics or ML) and can't qu…

For m = 1, the gradient is a "vector" (a column vector). The Jacobian is a functional/a linear map (a row vector, dual to a column vector). They're transposes of one another. For m > 1, I would normally just define the Jacobian as a linear map in the usual way and define the gradient to be its transpose. Remember that these are all just definitions at the end of the day and a little bit arbitrary.

Re: Matrix Calculus (For Machine Learning and Beyond)

#13

Post by jeffreyrogers »

windsignaling wrote: post #5

Great course. I highly recommended anyone interested in this topic to check it out on the MIT website, taught by the same authors. They are great lecturers.

Looks like the lectures from a prior version are on youtube too: https://www.youtube.com/playlist?list=PLUl4u3cNGP62EaLLH92E_...

Re: Matrix Calculus (For Machine Learning and Beyond)

#14

Post by Valk3_ »

i_am_proteus wrote: post #4

Those looking for a shorter primer could consult https://arxiv.org/abs/1802.01528

I've only skimmed through both of them, so I might be entirely incorrect here, but isn't the essential approach a bit different for both? The MIT one emphasis not to view matrices as tables of entries, but instead as holistic mathematical objects. So when they perform the derivatives, they try to avoid the "element-wise" approach of differentiation, while the one by Parr et Howard seems to do the "element-wise" approach, although with some shortcuts.

Re: Matrix Calculus (For Machine Learning and Beyond)

#15

Post by FilosofumRex »

wait what - another math textbook recommendation by academicians. ML and MLL are arts of tinkering not academic subjects.

Though Steven Johnson is the real deal and writes lots of code, Edelman is a shyster/imposter who used to ride the coattails of G. Strang and now shills for Julia where he makes most of his money. You don't need, and won't understand ML/LLM by reading textbooks.

1. If you want to have a little fun with ML/LLM, fire up Google Collab and run one of tutorials on the web - Karpathy, Hugging Face or PyTorch examples.

2. If you don't want to do, but just read for fun, Howard & Parr's essay as recommended by someone else here is much shorter and more succinct. https://explained.ai/matrix-calculus/ this link renders better

3. If you insist on academic textbooks, Boyd & Vandenberghe skips calculus and has more applications (engineering). Unfortunately, code examples are in Julia! https://web.stanford.edu/~boyd/vmls/vmls.pdf https://web.stanford.edu/~boyd/vmls/. link to python version

4. If you Want to become a tensor & differential programming ninja, learn Jax, XLA https://docs.jax.dev/en/latest/quickstart.html https://colab.research.google.com/github/exoplanet-dev/jaxop...

Re: Matrix Calculus (For Machine Learning and Beyond)

#16

Post by vismit2000 »

edflsafoiewq wrote: post #10
sfpotter wrote: post #8

If you want to get handy with matrix calculus, the real prerequisite is being comfortable with Taylor expansions and linear algebra. In a graduate numerical optimization class I took over a decade ago, the professor spent 10 minutes on the first day deriving some matrix calculus identity by working out the expressions for partial derivatives using simple calculus rules and a lot of manual labor. Then, as the class wa…

Can you give an example?

Check out this classic from 3b1b - How (and why) to raise e to the power of a matrix: https://youtu.be/O85OWBJ2ayo

Re: Matrix Calculus (For Machine Learning and Beyond)

#18

Post by godelski »

edflsafoiewq wrote: post #10
sfpotter wrote: post #8

If you want to get handy with matrix calculus, the real prerequisite is being comfortable with Taylor expansions and linear algebra. In a graduate numerical optimization class I took over a decade ago, the professor spent 10 minutes on the first day deriving some matrix calculus identity by working out the expressions for partial derivatives using simple calculus rules and a lot of manual labor. Then, as the class wa…

Can you give an example?

  https://math.stackexchange.com/questions/3680708/what-is-the-difference-between-the-jacobian-hessian-and-the-gradient

  https://carmencincotti.com/2022-08-15/the-jacobian-vs-the-hessian-vs-the-gradient/

Re: Matrix Calculus (For Machine Learning and Beyond)

#19

Post by sfpotter »

edflsafoiewq wrote: post #10
sfpotter wrote: post #8

If you want to get handy with matrix calculus, the real prerequisite is being comfortable with Taylor expansions and linear algebra. In a graduate numerical optimization class I took over a decade ago, the professor spent 10 minutes on the first day deriving some matrix calculus identity by working out the expressions for partial derivatives using simple calculus rules and a lot of manual labor. Then, as the class wa…

Can you give an example?

If you mean for how to use Taylor expansions and linear algebra, here's one I just made up.

Let's say I want to differentiate tr(X^T X), tr is the trace, X is a matrix, and X^T is its transpose. Expand:

    tr((X + dX)^T (X + dX)) = tr(X^T X) + 2 tr(X^T dX) + tr(dX^T dX).
Our knowledge of linear algebra tells us that tr is a linear map. Hence, dX -> 2 tr(X^T dX) is the linear mapping corresponding to the Jacobian of tr(X^T X). With a little more work we could figure out how to write it as a matrix.

Re: Matrix Calculus (For Machine Learning and Beyond)

#20

Post by godelski »

Valk3_ wrote: post #14
i_am_proteus wrote: post #4

Those looking for a shorter primer could consult https://arxiv.org/abs/1802.01528

I've only skimmed through both of them, so I might be entirely incorrect here, but isn't the essential approach a bit different for both? The MIT one emphasis not to view matrices as tables of entries, but instead as holistic mathematical objects. So when they perform the derivatives, they try to avoid the "element-wise" approach of differentiation, while the one by Parr et Howard seems to do the "element-wise" appro…

I got the same impression as you the Bright, Edelman, and Johnson (MIT) notes seems more driven my mathematicians where I find the Parr and Howard paper wanting. Though I agree with them

  >  Note that you do not need to understand this material before you start learning to train and use deep learning in practice
I have an alternative version

  > You don't need to know math to train good models, but you do need to know math to know why your models are wrong.
Referencing "All models are wrong"

I think another part is that the Bright, Edleman, and Johnson paper are also introducing concepts such as Automatic Differentiation, Root Finding, Finite Difference Methods, and ODEs. With that in mind it is far more important to be coming from the approach where you are understanding structures.

I think there is an odd pushback against math in the ML world (I'm a ML researcher). Mostly because it is hard and there's a lot of success you can gain without it. But I don't think that should discourage people from learning math. And frankly, the math is extremely useful. If we're ever going to understand these models we're going to need to do a fuck ton more math. So best to get started sooner than later (if that's anyone's personal goal anyways)

Post reply on HN