Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The matrix calculus you need for deep learning (2018)

arxiv.org

Post reply on HN
Print view View original

21–30 of 43 posts

Re: The matrix calculus you need for deep learning (2018)

#21

Post by yellowcake0 »

fn-mote wrote: post #14
yellowcake0 wrote: post #8

If you know single variable calculus and basic linear algebra, then you can often muddle through by just remembering that the derivative of, (1) f: R^n -> R is a vector (2) f: R -> R^n is a vector (3) f: R^m -> R^n is an n x m matrix (the Jacobian) (*) f(x)=x^tAx is f'(x)=(A + A^t)x (this is an example of (1)) (**) and that the derivative (gradient) of (1) gives you (3) with m=n, and in this case the derivative of (3…

I think that one of the reasons people look for a good exposition is so that they do not _have_ to muddle through. I know I look for a simple example or explanation that makes complicated cases clear. This 'explanation' might be correct, but I found the details that would help someone else follow it are missing. Especially someone with a computer science background who considers a m x 1 matrix different from a 1 x m…

I admit it's a blue collar approach, but it's served me well when my goal is optimization or some such and I just need to get on with it.

Re: The matrix calculus you need for deep learning (2018)

#22

Post by psalminen »

carlosf wrote: post #9
29athrowaway wrote: post #7

Earlier quoted context omitted.

To begin, try to get a really high level explanation for differential and integral calculus. Look at videos, illustrations, etc. Then, once you have a mental image of what you will be doing, get into the math from the ground up.

When I was like 14 years old one one my math teachers explained to us what calculus was using zero equations. He made everything look really easy. At college, it was the complete opposite. My professor started with limits and convergence, didn't even bother to explain why we were studying the subject. Kudos for people who actually explain things.

My high school physics teacher explained calculus to me by plotting a velocity curve, then saying the tangential line at a point (derivative) is acceleration and the area from 0 to that point under the curve (integral) was the distance travelled. Made all of calculus very easy for me to grok going forward.

Re: The matrix calculus you need for deep learning (2018)

#23

Post by p_m_c »

brailsafe wrote: post #3

> We assume no math knowledge beyond what you learned in calculus 1 Anyone got recommendations for self-studying and testing Calculus 1?

Calculus Made Easy - 1910. It is even entertaining.

"What One Fool Can Do, Another Can. (Ancient Simian Proverb.)"

https://calculusmadeeasy.org/

Re: The matrix calculus you need for deep learning (2018)

#25

Post by PEJOE »

yellowcake0 wrote: post #8

If you know single variable calculus and basic linear algebra, then you can often muddle through by just remembering that the derivative of, (1) f: R^n -> R is a vector (2) f: R -> R^n is a vector (3) f: R^m -> R^n is an n x m matrix (the Jacobian) (*) f(x)=x^tAx is f'(x)=(A + A^t)x (this is an example of (1)) (**) and that the derivative (gradient) of (1) gives you (3) with m=n, and in this case the derivative of (3…

I struggling in DL when we started having to do tensor multiplications on the homework. Good with 2D matrices but the higher dimensional math gave me a lot of trouble when it should have been straight forward.

Any tips there?

Re: The matrix calculus you need for deep learning (2018)

#26

Post by rdedev »

This helped me a lot in understanding the math behind gradient descent pretty well. The only problem I faced was when I tried to derive the equations for a vectorised implementation of a neural network. Certain derivatives, like the derivatives of matrix multiplication, are not obvious from the from the math presented in the paper. Had to dig through a bit to bridge that gap between non vectorised and vectorised implementations.

By vectorised implementation of gradient descent, I mean one where gradient descent looks at more that one training samplenat a time when updating weights

Re: The matrix calculus you need for deep learning (2018)

#28

Post by vishnugupta »

brailsafe wrote: post #3

> We assume no math knowledge beyond what you learned in calculus 1 Anyone got recommendations for self-studying and testing Calculus 1?

Better explained[1] articles that explain the “what” part, such as the meaning of “e”, from the first principles.

I have a reasonable handle on Calculus-1 but those articles really helped me connect the dots. After reading them I realized that all I knew was “how” without having a clue about “what”

[1] https://betterexplained.com/

Re: The matrix calculus you need for deep learning (2018)

#30

Post by beforeolives »

brailsafe wrote: post #3

> We assume no math knowledge beyond what you learned in calculus 1 Anyone got recommendations for self-studying and testing Calculus 1?

Nothing beats taking a good university class on it - https://ocw.mit.edu/courses/mathematics/18-01-single-variabl...
Post reply on HN