Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Matrix Calculus You Need for Deep Learning

explained.ai

Post reply on HN
Print view View original

61–70 of 79 posts

Re: The Matrix Calculus You Need for Deep Learning

#61

Post by tachyonbeam »

RobertDeNiro wrote: post #57
hypersoar wrote: post #37

Does anyone know of good resources for studying machine learning or data science given a strong mathematical background? I'm transitioning careers from pure math research into industry. I know very little about machine learning, but I know the crap out of linear algebra and real analysis (and other, less relevant fields). It'd be great to have some sources that leverage that without assuming much prior CS knowledge.

You don't need much CS knowledge for deep learning, in fact pure math/stats is more useful. I'd say CS knowledge is only useful if you do RL stuff and even then you don't really need it that much.

That really depends on what you're doing. I've seen people coming from a pure math background really struggle. If you've never programmed before, then using something like TensowFlow isn't exactly going to feel natural. Familiarity with Python and the Linux ecosystem would definitely be quite helpful. Yes, I'm sure you could run ML models on a Windows machine, but at some point, you might want to perform experiments on a cluster, and it's going to be running Linux.

Re: The Matrix Calculus You Need for Deep Learning

#62

Post by ilzmastr »

hypersoar wrote: post #37

Does anyone know of good resources for studying machine learning or data science given a strong mathematical background? I'm transitioning careers from pure math research into industry. I know very little about machine learning, but I know the crap out of linear algebra and real analysis (and other, less relevant fields). It'd be great to have some sources that leverage that without assuming much prior CS knowledge.

Are you only interested in ML practicum? What did you take after real analysis?

What are your thoughts/interests on analysis for ML, like the approximation theory that branched from Fourier and Wavelet analysis catching on with Cybenko [1], continuing with among others Mhaskar [2], most recently added to by Bolcskei [3]. And then there are other areas where analysis applies like studying the optimization of such networks...

(And of course, this is just for NNs. There are other areas of research where analysis comes into ML. And of course, real analysis lays the foundations for probability and statistics, and is not abstracted away in many research areas in these fields.)

[1](https://pdfs.semanticscholar.org/05ce/b32839c26c8d2cb38d5529...) [2](https://pdfs.semanticscholar.org/694a/d455c119c0d07036792b80...) [3](https://arxiv.org/abs/1705.01714)

Re: The Matrix Calculus You Need for Deep Learning

#63

Post by mindviews »

hypersoar wrote: post #37

Does anyone know of good resources for studying machine learning or data science given a strong mathematical background? I'm transitioning careers from pure math research into industry. I know very little about machine learning, but I know the crap out of linear algebra and real analysis (and other, less relevant fields). It'd be great to have some sources that leverage that without assuming much prior CS knowledge.

For Machine Learning, I found "Learning From Data" http://amlbook.com/ to be a very strong, foundational book. It does the best job of any reference I've seen of answering the question: "Is learning possible and feasible?" Provides and excellent mathematical foundation for understanding when and why ML systems should succeed and fail. Associated MOOC from Caltech at https://work.caltech.edu/telecourse.html

Re: The Matrix Calculus You Need for Deep Learning

#64

Post by nightski »

man-and-laptop wrote: post #59
nightski wrote: post #49

Earlier quoted context omitted.

It really is a great book. However my friend & I attempted to make it through and faced quite a bit of struggle. Eventually we'd figure things out but it felt like it would of been significantly easier with the help of a teacher/mentor to ask questions to. Unfortunately it's actually kind of hard to find colleges nearby that even have courses in Bayesian statistics (from an inference perspective). That was frustratin…

It's far from state of the art, but reading Computer Vision by Simon J D Prince [1] alongside David Barber's BRML can really help. [1] - http://www.computervisionmodels.com/

I had this book on my computer back from when it was released, but never got around to reading it. Thank you for the recommendation!

Re: The Matrix Calculus You Need for Deep Learning

#65

Post by tzahola »

thanatropism wrote: post #45
hypersoar wrote: post #37

Does anyone know of good resources for studying machine learning or data science given a strong mathematical background? I'm transitioning careers from pure math research into industry. I know very little about machine learning, but I know the crap out of linear algebra and real analysis (and other, less relevant fields). It'd be great to have some sources that leverage that without assuming much prior CS knowledge.

There are many new methods cropping up that most people in the data science hype train will be full-on unable to access, including methods on manifolds (even kernel methods on manifolds) and algebraic-topological methods (persistent homology) with enough maths to give Kagglers the screaming meemies. I'm using some of those for $(redacted, the plan is to make money). Don't follow the crowd.

Math? Meh.

Just throw more GPUs at it bro. Gradient descent 4ever! xD

/s

Re: The Matrix Calculus You Need for Deep Learning

#66

Post by cs702 »

hypersoar wrote: post #37

Does anyone know of good resources for studying machine learning or data science given a strong mathematical background? I'm transitioning careers from pure math research into industry. I know very little about machine learning, but I know the crap out of linear algebra and real analysis (and other, less relevant fields). It'd be great to have some sources that leverage that without assuming much prior CS knowledge.

If you have at least some coding experience and you are interested in the practical aspects of ML/DL (i.e., you want to learn the how-to, not the why or the whence), my recommendation is to start with the fast.ai courses by Jeremy Howard (co-author of this "Matrix Calculus" cheat sheet) and Rachel Thomas[a]:

* fast.ai ML course: http://forums.fast.ai/t/another-treat-early-access-to-intro-...

* fast.ai DL course: part 1: http://course.fast.ai/ part 2: http://course.fast.ai/part2.html

The fast.ai courses spend very little time on theory, and you can follow the videos at your own pace.

Books:

* The best books on ML (excluding DL), in my view, are "An Introduction to Statistical Learning" by James, Witten, Hastie and Tibshirani, and "The Elements of Statistical Learning" by Hastie, Tibshirani and Friedman. The Elements arguably belongs on every ML practitioner's bookshelf -- it's a fantastic reference manual.[b]

* The only book on DL that I'm aware of is "Deep Learning," by Goodfellow, Bengio and Courville. It's a good book, but I suggest holding off on reading it until you've had a chance to experiment with a range of deep learning models. Otherwise, you will get very little useful out of it.[c]

Good luck!

[a] Scroll down on this page for their bios: http://course.fast.ai/about.html

[b] Introduction to Statistical Learning: http://www-bcf.usc.edu/~gareth/ISL/ The Elements of Statistical Learning: https://web.stanford.edu/~hastie/ElemStatLearn/

[c] http://www.deeplearningbook.org/

Re: The Matrix Calculus You Need for Deep Learning

#67

Post by utk09 »

hypersoar wrote: post #37

Does anyone know of good resources for studying machine learning or data science given a strong mathematical background? I'm transitioning careers from pure math research into industry. I know very little about machine learning, but I know the crap out of linear algebra and real analysis (and other, less relevant fields). It'd be great to have some sources that leverage that without assuming much prior CS knowledge.

https://github.com/llSourcell/Learn_Machine_Learning_in_3_Mo...

This is also a good resource for MachineLearning

Re: The Matrix Calculus You Need for Deep Learning

#68

Post by soVeryTired »

bllguo wrote: post #58
soVeryTired wrote: post #55

Earlier quoted context omitted.

Honestly, I don't understand the way he explains things. The maths is difficult to follow, and it just never clicks for me. Maybe he's writing for someone with a physics background or something, but I feel stupid when I read bishop. I just read over his description of how to transform a uniform random variable into a variable with a desired distribution (p. 526). It's a fairly easy trick, but if I didn't already know…

I'm trying to read through it and I have to agree, his math isn't that clear to me. What do you recommend?

David Barber!

Re: The Matrix Calculus You Need for Deep Learning

#69

Post by saamm »

mindviews wrote: post #63
hypersoar wrote: post #37

Does anyone know of good resources for studying machine learning or data science given a strong mathematical background? I'm transitioning careers from pure math research into industry. I know very little about machine learning, but I know the crap out of linear algebra and real analysis (and other, less relevant fields). It'd be great to have some sources that leverage that without assuming much prior CS knowledge.

For Machine Learning, I found "Learning From Data" http://amlbook.com/ to be a very strong, foundational book. It does the best job of any reference I've seen of answering the question: "Is learning possible and feasible?" Provides and excellent mathematical foundation for understanding when and why ML systems should succeed and fail. Associated MOOC from Caltech at https://work.caltech.edu/telecourse.html

My ML professor used this book, and I loved it. Some presentations of ML make it seem like a bag of tricks; this feels like the opposite of that.

Re: The Matrix Calculus You Need for Deep Learning

#70

Post by srean »

hypersoar wrote: post #37

Does anyone know of good resources for studying machine learning or data science given a strong mathematical background? I'm transitioning careers from pure math research into industry. I know very little about machine learning, but I know the crap out of linear algebra and real analysis (and other, less relevant fields). It'd be great to have some sources that leverage that without assuming much prior CS knowledge.

Given your math bent expecially analysis, I would recommend you start with Luc Devroye, follow it up with Vapnik. These are a bit dated but still very relevant. After these two go with Alex Smola and Bernard Scholkopf. The books that have been suggested to you so far aren't bad but with high fluff to meat ratio, not meant for a reader like you. If you have function analysis in your bag you will feel right at home with the books I suggested. Deep NN are much in the news, but on the math side of the practice of DNN, there is not much to it apart from chain rule for differentiation.
Post reply on HN