Ask HN: What maths are critical to pursuing ML/AI? • Hacker News Board

Re: Ask HN: What maths are critical to pursuing ML/AI?

#31

Post by dtjon » Mon, Aug 28, 2017, 3:41 PM UTC

digitalzombie wrote: post #26

CuriouslyC wrote: post #2

You absolutely need a solid grounding in multi-variable calculus, linear algebra, probability theory and information theory. It will also be helpful to be well versed in graph theory. In my opinion one of the best starting points is "Information Theory, Inference and Learning Algorithms" by David MacKaye. It's a bit long in the tooth now, but it is still one of the most approachable and well written books in the fiel…

> "Bayesian Data Analysis" by Andrew Gelman is another great read. If you want to read that book you need real analysis more specifically measure theory (unless that subject is in probability theory for you). You cannot get into the last few chapters without it. Dirichlet Process are described using measures. I don't believe you need multivar calc or info theory. Info theory stuff are used but not as often. I believe…

Great class and great professor. One of my favorite classes from my degree.

Re: Ask HN: What maths are critical to pursuing ML/AI?

#32

Post by jey » Mon, Aug 28, 2017, 3:41 PM UTC

tptacek wrote: post #20

CuriouslyC wrote: post #2

You absolutely need a solid grounding in multi-variable calculus, linear algebra, probability theory and information theory. It will also be helpful to be well versed in graph theory. In my opinion one of the best starting points is "Information Theory, Inference and Learning Algorithms" by David MacKaye. It's a bit long in the tooth now, but it is still one of the most approachable and well written books in the fiel…

I'm not super interested in ML but I am very interested in applied mathematics in computer science. I've got a fair bit of linear algebra due to cryptography, but have had virtually no need of any form of calculus (unless I'm relying on it without knowing it) in my career. So beyond just saying that you'd need grounding in multivariable calculus to do serious ML work, I would be super interested in hearing more about…

All of statistics and machine learning involves lots of integrals and derivatives. For example: expected values are integrals, and model fitting is done by hill climbing in the direction of the derivative.

Re: Ask HN: What maths are critical to pursuing ML/AI?

#33

Post by mindcrime » Mon, Aug 28, 2017, 3:41 PM UTC

tptacek wrote: post #20

CuriouslyC wrote: post #2

You absolutely need a solid grounding in multi-variable calculus, linear algebra, probability theory and information theory. It will also be helpful to be well versed in graph theory. In my opinion one of the best starting points is "Information Theory, Inference and Learning Algorithms" by David MacKaye. It's a bit long in the tooth now, but it is still one of the most approachable and well written books in the fiel…

I'm not super interested in ML but I am very interested in applied mathematics in computer science. I've got a fair bit of linear algebra due to cryptography, but have had virtually no need of any form of calculus (unless I'm relying on it without knowing it) in my career. So beyond just saying that you'd need grounding in multivariable calculus to do serious ML work, I would be super interested in hearing more about…

The most obvious thing is understanding back-propagation. Backprop is pretty much all partial derivatives / chain rule manipulations. Also a lot of machine learning involves convex optimization which entails some calculus.

Re: Ask HN: What maths are critical to pursuing ML/AI?

#34

Post by fnbr » Mon, Aug 28, 2017, 3:43 PM UTC

tptacek wrote: post #20

CuriouslyC wrote: post #2

You absolutely need a solid grounding in multi-variable calculus, linear algebra, probability theory and information theory. It will also be helpful to be well versed in graph theory. In my opinion one of the best starting points is "Information Theory, Inference and Learning Algorithms" by David MacKaye. It's a bit long in the tooth now, but it is still one of the most approachable and well written books in the fiel…

I'm not super interested in ML but I am very interested in applied mathematics in computer science. I've got a fair bit of linear algebra due to cryptography, but have had virtually no need of any form of calculus (unless I'm relying on it without knowing it) in my career. So beyond just saying that you'd need grounding in multivariable calculus to do serious ML work, I would be super interested in hearing more about…

A lot of probability theory requires it. For instance, ML is largely framed mathematically as a series of optimisation problem, which are then solved by finding the gradient and performing gradient descent; this requires elementary calculus to calculate the gradient.

Additionally, if you want to calculate a probability given a density function, or evaluate an expectation, you need to calculate several integrals. This arises quite often in the theoretical sections of ML papers/textbooks.

The use of calculus in ML is probably similar to the use of number theory in crypto- you can do applied work fine without it, but you understand the work a lot better by knowing the math, and are less likely to make dumb mistakes.

Re: Ask HN: What maths are critical to pursuing ML/AI?

#35

Post by jules » Mon, Aug 28, 2017, 3:43 PM UTC

* Calculus

* Linear algebra

* Optimisation

* Probability

Various universities have very good course content freely available online, often including textbook recommendations, course notes, exercises, sample exams, and video lectures. Realistically it is probably going to be quite difficult to learn this on your own.

Re: Ask HN: What maths are critical to pursuing ML/AI?

#36

Post by dtjon » Mon, Aug 28, 2017, 3:44 PM UTC

CuriouslyC wrote: post #2

You absolutely need a solid grounding in multi-variable calculus, linear algebra, probability theory and information theory. It will also be helpful to be well versed in graph theory. In my opinion one of the best starting points is "Information Theory, Inference and Learning Algorithms" by David MacKaye. It's a bit long in the tooth now, but it is still one of the most approachable and well written books in the fiel…

+1 for "Elements of Statistical Learning", this is the basis for most rigorous intro to ML classes

Re: Ask HN: What maths are critical to pursuing ML/AI?

#37

Post by jules » Mon, Aug 28, 2017, 3:44 PM UTC

dtjon wrote: post #29

Probability, and thus multivariate calculus and partial differential equations. Linear algebra. Convex Optimization, and thus multivariate and partial differential equations. Some principals of statistics is usually helpful

Why do you need partial differential equations? I don't think you necessarily need any knowledge of differential equations to do ML, though the top ML people certainly would know it because of their general math education.

Re: Ask HN: What maths are critical to pursuing ML/AI?

#38

Post by [deleted] » Mon, Aug 28, 2017, 3:45 PM UTC

tptacek wrote: post #20

CuriouslyC wrote: post #2

You absolutely need a solid grounding in multi-variable calculus, linear algebra, probability theory and information theory. It will also be helpful to be well versed in graph theory. In my opinion one of the best starting points is "Information Theory, Inference and Learning Algorithms" by David MacKaye. It's a bit long in the tooth now, but it is still one of the most approachable and well written books in the fiel…

I'm not super interested in ML but I am very interested in applied mathematics in computer science. I've got a fair bit of linear algebra due to cryptography, but have had virtually no need of any form of calculus (unless I'm relying on it without knowing it) in my career. So beyond just saying that you'd need grounding in multivariable calculus to do serious ML work, I would be super interested in hearing more about…

[deleted]

Re: Ask HN: What maths are critical to pursuing ML/AI?

#39

Post by jules » Mon, Aug 28, 2017, 3:47 PM UTC

popcorncolonel wrote: post #7

CuriouslyC wrote: post #2

You absolutely need a solid grounding in multi-variable calculus, linear algebra, probability theory and information theory. It will also be helpful to be well versed in graph theory. In my opinion one of the best starting points is "Information Theory, Inference and Learning Algorithms" by David MacKaye. It's a bit long in the tooth now, but it is still one of the most approachable and well written books in the fiel…

I disagree that you need a solid founding in information theory. Almost all that I've seen about IT in ML is minimizing the KL divergence, which can be learned by browsing the wiki page.

Well, information theory isn't much more than the logarithm of probability theory, so it doesn't hurt to learn it anyway. The only thing you need to know is that given a probability distribution P there exist a compression scheme to encode a value X with a message of P_length(X) = log(1/P(X)) bits. This can be summarised as BITS = log(1/PROBABILITY). Entropy is just the average number of bits you need to encode a random value from distribution P with the compression scheme of distribution P, i.e. E_P[P_length(X)]. The KL(P,Q) divergence is when you encode a random value from distribution P with the compression scheme of distribution Q. Say you're compressing english text but you're using a compressor tailored to spanish. The KL divergence is how many extra bits you need (on average) compared to encoding the english text with the english compressor:

KL(P,Q) = E_P[Q_length(X)] - E_P[P_length(X)]

Re: Ask HN: What maths are critical to pursuing ML/AI?

#40

Post by CuriouslyC » Mon, Aug 28, 2017, 3:48 PM UTC

tptacek wrote: post #20

CuriouslyC wrote: post #2

You absolutely need a solid grounding in multi-variable calculus, linear algebra, probability theory and information theory. It will also be helpful to be well versed in graph theory. In my opinion one of the best starting points is "Information Theory, Inference and Learning Algorithms" by David MacKaye. It's a bit long in the tooth now, but it is still one of the most approachable and well written books in the fiel…

I'm not super interested in ML but I am very interested in applied mathematics in computer science. I've got a fair bit of linear algebra due to cryptography, but have had virtually no need of any form of calculus (unless I'm relying on it without knowing it) in my career. So beyond just saying that you'd need grounding in multivariable calculus to do serious ML work, I would be super interested in hearing more about…

The optimization techniques required to actually fit models are almost all powered by some form of gradient descent, and integration is usually required in truly probabilistic models to go from a density function to predictions.