Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The matrix calculus you need for deep learning (2018)

arxiv.org

Post reply on HN
Print view View original

11–20 of 43 posts

Re: The matrix calculus you need for deep learning (2018)

#11

Post by rademacher »

yellowcake0 wrote: post #8

If you know single variable calculus and basic linear algebra, then you can often muddle through by just remembering that the derivative of, (1) f: R^n -> R is a vector (2) f: R -> R^n is a vector (3) f: R^m -> R^n is an n x m matrix (the Jacobian) (*) f(x)=x^tAx is f'(x)=(A + A^t)x (this is an example of (1)) (**) and that the derivative (gradient) of (1) gives you (3) with m=n, and in this case the derivative of (3…

I suggest this reference [1] for anyone needing to look something up.

[1] https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf

Re: The matrix calculus you need for deep learning (2018)

#12

Post by benrbray »

yellowcake0 wrote: post #8

If you know single variable calculus and basic linear algebra, then you can often muddle through by just remembering that the derivative of, (1) f: R^n -> R is a vector (2) f: R -> R^n is a vector (3) f: R^m -> R^n is an n x m matrix (the Jacobian) (*) f(x)=x^tAx is f'(x)=(A + A^t)x (this is an example of (1)) (**) and that the derivative (gradient) of (1) gives you (3) with m=n, and in this case the derivative of (3…

I think it conceptually helps a lot to distinguish between row and column vectors, even if they are coalesce in any production code you might write.

(0a) An (nxm) matrix A represents a linear transformation f(x)=Ax from R^m -> R^n

(0b) A linear transformation f(x) = Ax is its own derivative, f'(x) = Ax

(3) The derivative of a function f : R^m -> R^n is a linear transformation with the same "type" R^m -> R^n as the original function. This means it is an (nxm) matrix.

The other two rules can be derived from these rules:

(1) The derivative of a function f : R^m -> R is a (1xm) matrix, aka a row vector.

(2) The derivative of a function f : R^n -> R is an (nx1) matrix, aka a column vector.

This perspective is helpful because, aided by the chain rule, we can easily derive all the other matrix algebra rules on the fly. Distinguishing row / column vectors helps catch conceptual type errors in places where it's inappropriate to add vectors and covectors together.

Re: The matrix calculus you need for deep learning (2018)

#14

Post by fn-mote »

yellowcake0 wrote: post #8

If you know single variable calculus and basic linear algebra, then you can often muddle through by just remembering that the derivative of, (1) f: R^n -> R is a vector (2) f: R -> R^n is a vector (3) f: R^m -> R^n is an n x m matrix (the Jacobian) (*) f(x)=x^tAx is f'(x)=(A + A^t)x (this is an example of (1)) (**) and that the derivative (gradient) of (1) gives you (3) with m=n, and in this case the derivative of (3…

I think that one of the reasons people look for a good exposition is so that they do not _have_ to muddle through.

I know I look for a simple example or explanation that makes complicated cases clear.

This 'explanation' might be correct, but I found the details that would help someone else follow it are missing. Especially someone with a computer science background who considers a m x 1 matrix different from a 1 x m matrix.

I see a sister comment has explained this, but I want to point out that there are plenty of details... (like sister's statement 0b which looks to me more like an exponential function than a linear function) that's why one goes to a carefully written source.

This paper is a carefully written explanation of the basics, ending with many applications of the chain rule. The "matrix product rule" is not mentioned, which is good in my opinion.

Re: The matrix calculus you need for deep learning (2018)

#15

Post by mhh__ »

brailsafe wrote: post #3

> We assume no math knowledge beyond what you learned in calculus 1 Anyone got recommendations for self-studying and testing Calculus 1?

I found the book "Calculus: a complete course" very nice. It's very cheap and extremely unpretentious while also giving you a subtle taste of a bunch of interesting applications (Deep learning... bleh, give me electromagnetism!)

Re: The matrix calculus you need for deep learning (2018)

#16

Post by ABeeSea »

brailsafe wrote: post #3

> We assume no math knowledge beyond what you learned in calculus 1 Anyone got recommendations for self-studying and testing Calculus 1?

They are probably terrible for self-study, but I want to mention them anyway. I am a huge fan of David Bressoud’s Calculus (and analysis!) books. His first one doesn’t have exercises and his second one has a lot of physics.

They are all heavy on written narrative and interlaced with history. I find all four of them absolutely fascinating.

Calculus: https://www.amazon.com/Calculus-Reordered-History-Big-Ideas/...

https://www.amazon.com/Second-Year-Calculus-Undergraduate-Ma...

Analysis:

https://www.amazon.com/Approach-Analysis-Mathematical-Associ...

https://www.amazon.com/Lebesgues-Integration-Mathematical-As...

Re: The matrix calculus you need for deep learning (2018)

#18

Post by 29athrowaway »

carlosf wrote: post #9
29athrowaway wrote: post #7

Earlier quoted context omitted.

To begin, try to get a really high level explanation for differential and integral calculus. Look at videos, illustrations, etc. Then, once you have a mental image of what you will be doing, get into the math from the ground up.

When I was like 14 years old one one my math teachers explained to us what calculus was using zero equations. He made everything look really easy. At college, it was the complete opposite. My professor started with limits and convergence, didn't even bother to explain why we were studying the subject. Kudos for people who actually explain things.

I never got a high level explanation for what calculus was because the business model of my school involved having a passing rate below 30%. If the passing rate was too high, teachers got fired no matter what the reason was.

Re: The matrix calculus you need for deep learning (2018)

#19

Post by dang »

If curious, past threads:

Matrix calculus for deep learning part 2 - https://news.ycombinator.com/item?id=23358761 - May 2020 (6 comments)

Matrix Calculus for Deep Learning - https://news.ycombinator.com/item?id=21661545 - Nov 2019 (47 comments)

The Matrix Calculus You Need for Deep Learning - https://news.ycombinator.com/item?id=17422770 - June 2018 (77 comments)

Matrix Calculus for Deep Learning - https://news.ycombinator.com/item?id=16267178 - Jan 2018 (81 comments)

Others?

Re: The matrix calculus you need for deep learning (2018)

#20

Post by xzel »

cinntaile wrote: post #17
airocker wrote: post #6

is there an online course for maths? like undergrad maths?

https://tutorial.math.lamar.edu/ are pretty good, although they're mostly focused on the applied part.

Paul's notes are great! Carried my butt through Calc 3 and PDEs, especially when the math for these classes get a little more formulaic (that is there is a pretty small set of reasonable questions you could possibly ask). His stuff is significantly more well done than any text book I had assigned for college math.
Post reply on HN