Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Matrix Calculus You Need for Deep Learning

explained.ai

Post reply on HN
Print view View original

41–50 of 79 posts

Re: The Matrix Calculus You Need for Deep Learning

#41

Post by soVeryTired »

hypersoar wrote: post #37

Does anyone know of good resources for studying machine learning or data science given a strong mathematical background? I'm transitioning careers from pure math research into industry. I know very little about machine learning, but I know the crap out of linear algebra and real analysis (and other, less relevant fields). It'd be great to have some sources that leverage that without assuming much prior CS knowledge.

For non-deep learning, read David Barber's book:

http://web4.cs.ucl.ac.uk/staff/D.Barber/textbook/090310.pdf

Some sections may be less relevant, depending on what you want to do, but Section III is a very good introduction to machine learning methods.

Do the exercises as you're reading. Theory is one thing, but in ML my rule of thumb is that you don't really understand a model until you've coded it up. A collection of written exercises would be a good way to impress an interviewer, too.

Re: The Matrix Calculus You Need for Deep Learning

#42

Post by soVeryTired »

mlthoughts2018 wrote: post #40
hypersoar wrote: post #37

Does anyone know of good resources for studying machine learning or data science given a strong mathematical background? I'm transitioning careers from pure math research into industry. I know very little about machine learning, but I know the crap out of linear algebra and real analysis (and other, less relevant fields). It'd be great to have some sources that leverage that without assuming much prior CS knowledge.

I made the same transition earlier in my career. One book on deep learning that meets your requirements is [0]. It’s readable, covers a broad set of modern topics, and has pragmatic tips for real use cases. For general machine learning, there are many, many books. A good intro is [1] and a more comprehensive, reference sort of book is [2]. Frankly, by this point, even reading the documentation and user guide of sciki…

+1 for Gelman, but I hate Bishop's book [2]. It was an early go-to reference in the field, but there are better books out there now.

Re: The Matrix Calculus You Need for Deep Learning

#43

Post by Sinidir »

ghaff wrote: post #25
TJSomething wrote: post #9

Earlier quoted context omitted.

Those Wikipedia pages are kind of awful for pedagogy, but they have the right equations, so I won't cover those. Say we have a curve that corresponds to how good of a fit your model is. We want to try to find the maximum on that curve. However, calculating every point of the curve is too expensive, so we want to minimize the number of points we have to check. So, we start with a guess as to the highest point on the c…

Unfortunately, mathematics is one of the areas where Wikipedia is pretty awful in general. The articles seem mostly written for people who pretty much already understand the topic in question. Of course, you always have to assume some knowledge base but the stereotypical jargon-filled Wilipedia approach is particularly off-putting in this area.

My uni lectures used to be in the same style as wiki articles. Professor reading from a extremely dense abstract script every lecture and not really explaining anything.

Bad times.

Re: The Matrix Calculus You Need for Deep Learning

#44

Post by Twisol »

archgoon wrote: post #11
roel_v wrote: post #2

So I have a question somewhat related to this that I never knew where/who to ask (well actually I asked a few mathematicians at a university I work with whose answers I couldn't understand - their answers were almost as impenetrable as the Wikipedia page, and some engineering scientists who I thought would be more into 'applied math' but they didn't know. So I'm hoping some data science people reading this would bett…

So, when you perform a regression of any sort, what you're doing is saying "Hey, I want to find parameters X,Y,Z, etc, that make this curve best fit the data that I have". One interpretation of 'best fit' is 'minimize the mean squared error'. So regression is just a minimization problem. You're trying to find the values that minimize f(X,Y,Z...) And, well, that means that you just want to find values for X,Y,Z such t…

This is really a fantastic explanation, but I want to clarify one thing from early on for other readers. If the function isn't changing much locally, it's possible that you are at neither a minimum or a maximum, but rather a saddle point. For instance, every function f(x) = x^k for (positive) odd k has a saddle point at x=0. Both the first and second derivative are 0 at saddle points -- another reason we might be interested in the second derivative and its kin.

Re: The Matrix Calculus You Need for Deep Learning

#45

Post by thanatropism »

hypersoar wrote: post #37

Does anyone know of good resources for studying machine learning or data science given a strong mathematical background? I'm transitioning careers from pure math research into industry. I know very little about machine learning, but I know the crap out of linear algebra and real analysis (and other, less relevant fields). It'd be great to have some sources that leverage that without assuming much prior CS knowledge.

There are many new methods cropping up that most people in the data science hype train will be full-on unable to access, including methods on manifolds (even kernel methods on manifolds) and algebraic-topological methods (persistent homology) with enough maths to give Kagglers the screaming meemies.

I'm using some of those for $(redacted, the plan is to make money). Don't follow the crowd.

Re: The Matrix Calculus You Need for Deep Learning

#47

Post by mlevental »

mxwsn wrote: post #32
mlevental wrote: post #30

Earlier quoted context omitted.

what is the name of the first method (fit parabolic surface)?

In optimization it's known as Newton's method. See the third section here for an intuitive image of repeated parabola-fitting. https://ardianumam.wordpress.com/2017/09/27/newtons-method-o... Wiki: https://en.wikipedia.org/wiki/Newton%27s_method_in_optimizat...

i feel silly for never realizing before that this was the appropriate geometric interpretation of newton's method.

Re: The Matrix Calculus You Need for Deep Learning

#48

Post by hikarudo »

soVeryTired wrote: post #42
mlthoughts2018 wrote: post #40

Earlier quoted context omitted.

I made the same transition earlier in my career. One book on deep learning that meets your requirements is [0]. It’s readable, covers a broad set of modern topics, and has pragmatic tips for real use cases. For general machine learning, there are many, many books. A good intro is [1] and a more comprehensive, reference sort of book is [2]. Frankly, by this point, even reading the documentation and user guide of sciki…

+1 for Gelman, but I hate Bishop's book [2]. It was an early go-to reference in the field, but there are better books out there now.

What do you hate about Bishop's book? I'm genuinely curious.

Re: The Matrix Calculus You Need for Deep Learning

#49

Post by nightski »

soVeryTired wrote: post #41
hypersoar wrote: post #37

Does anyone know of good resources for studying machine learning or data science given a strong mathematical background? I'm transitioning careers from pure math research into industry. I know very little about machine learning, but I know the crap out of linear algebra and real analysis (and other, less relevant fields). It'd be great to have some sources that leverage that without assuming much prior CS knowledge.

For non-deep learning, read David Barber's book: http://web4.cs.ucl.ac.uk/staff/D.Barber/textbook/090310.pdf Some sections may be less relevant, depending on what you want to do, but Section III is a very good introduction to machine learning methods. Do the exercises as you're reading. Theory is one thing, but in ML my rule of thumb is that you don't really understand a model until you've coded it up. A collection o…

It really is a great book. However my friend & I attempted to make it through and faced quite a bit of struggle. Eventually we'd figure things out but it felt like it would of been significantly easier with the help of a teacher/mentor to ask questions to. Unfortunately it's actually kind of hard to find colleges nearby that even have courses in Bayesian statistics (from an inference perspective). That was frustrating.

Re: The Matrix Calculus You Need for Deep Learning

#50

Post by Iwan-Zotow »

mlthoughts2018 wrote: post #40
hypersoar wrote: post #37

Does anyone know of good resources for studying machine learning or data science given a strong mathematical background? I'm transitioning careers from pure math research into industry. I know very little about machine learning, but I know the crap out of linear algebra and real analysis (and other, less relevant fields). It'd be great to have some sources that leverage that without assuming much prior CS knowledge.

I made the same transition earlier in my career. One book on deep learning that meets your requirements is [0]. It’s readable, covers a broad set of modern topics, and has pragmatic tips for real use cases. For general machine learning, there are many, many books. A good intro is [1] and a more comprehensive, reference sort of book is [2]. Frankly, by this point, even reading the documentation and user guide of sciki…

Goodfellow book [0] is available for free, http://www.deeplearningbook.org/
Post reply on HN