Learning Math for Machine Learning

Might look at the thread

Foundations Machine Learning (bloomberg.github.io)

at

https://news.ycombinator.com/item?id=17519591

There machine learning (ML) is basically a lot of empirical curve fitting. The context is usually with a lot of data, thousands of variables, millions or billions of data points, observations, pairs of values of thousands of independent variables and the value of the corresponding dependent variable. The work is all a larger, more data, version of: You have a high school style X-Y coordinate system and some points plotted there. So, you want to find values for coefficients a and b so the line

y = ax + b

fits the points as well as possible. But, you can do variations, try to fit, say,

log(y) = a sin(x) + b

Or replace log or sin with any functions you want and try again.

The logic, rational support, is essentially as follows: So, take, say, 1000 x-y pairs. Partition these into 500 training data and 500 test data. Find the best fit you can, using whatever fits, to the training data. Then take the equation and see how well it fits the test data. If the fit of the test data is also good, then that is your model.

Now you want to apply the model in practice, apply the model to data did not see in the given 1000 points. So for the application, will be given a value of x, plug it into the equation, and get the corresponding value of y. That's what you want -- maybe the value of y gives you Y|N for ad targeting, Y|N cancer, what MSFT will be selling for next month, what the revenue will be for next year, etc.

The rational, logical justification here is an assumption (which should have some justification from somewhere) that the x you are given and the y you want for that value of x is sufficiently like the x-y values you had in the original 1000 points.

Okay. Empirical curve fitting to a lot of data to make a predictive model, that is found with training data, tested with test data, and applied where the given data in the application is like the data used in the fitting.

The OP mentions that some people believe that to make progress to real machine intelligence, need more math than what I outlined.

My guess is that to make that intended progress, for all but some tiny niche cases, first need some much more powerful and quite different ideas, techniques, etc. than in the curve fitting ML I outlined.

Yes, there is a chance that with lots of data from working brains and lots of such empirical fitting we will be able to find some fits that will uncover some of the workings of the brain crucial for real intelligence. Uh, that's a definite maybe!

But there is a lot more to what can be done to build predictive models than such curve fitting, empirical or otherwise. I outlined some such in the thread that I referenced above.

So, for the question in the OP, what math? Well, if want to pursue directions other than the empirical curve fitting in the Bloomberg course I referenced above, my experience is -- quite a lot. For the education, start with a good undergraduate major in pure math. So, cover the usual topics, calculus, abstract algebra, linear algebra, differential equations, advanced calculus, probability, statistics. Then continue with more in algebra, analysis, and geometry.

ya lol and Hamiltonians. sometimes people just reel off all the math they've heard of to sound impressive. next we'll have people talking about de rham cohomology because of TDA (or something like that)

I'm kinda curious why so many people think that Linear Algebra Done Right is an introductory book for beginners who have math anxiety. Don't get me wrong, the book is great and I enjoyed working it through. It was a magical experience when I saw how simple it was to prove some seemingly hard theorems by just linking the right definitions and theorems. That said, the book does require certain level of math maturity as it achieves its elegance by staying at certain level of abstraction and its style is quite formal, so much so that a person who can use this book as its first linear algebra textbook shouldn't have math anxiety at all.

Speaking as one of the people who recommended it in this thread: I don't think math anxiety is the right focus for which textbook to choose. More precisely, I don't think you should try to solve that problem by getting a different linear algebra textbook. To put it bluntly, someone with math anxiety probably just doesn't have the mathematical maturity for linear algebra yet. In that case they'd be doing themselves a disservice by attempting the material using some sort of "more accessible" book; instead, they should focus on resolving that anxiety through developing a solid foundation in the prerequisite material.

Linear Algebra is typically the first course in which students have to transition from predominantly rote computation to proof-based theory. Axler's Linear Algebra Done Right is very often the textbook used for that course because it (mostly [1]) lives up to its name. This isn't Math 55: compared to Rudin and Halmos, Axler is a very accessible introduction to linear algebra for those who are ready for linear algebra. The floor for understanding this subject doesn't doesn't get much lower than Axler (and in my opinion, it doesn't get much better at the undergraduate level either).

It's unfortunate that so many people want to skip to math they're not ready for, because there's no shame in building up to it. A lot of frustration can be eliminated by figuring out what you're actually prepared for and starting from there. If that means reviewing high school algebra then so be it; better to review "easy" material than to bounce around a dozen resources for advanced material you're not ready for.

__________________

1. See Noam Elkies' commentary on where it could improve: http://www.math.harvard.edu/~elkies/M55a.10/index.html

Hamiltonian mechanics, along with many other seemingly out of place 'advanced' maths, show up in modern Bayesian statistics pretty frequently. Hamiltonian Monte Carlo/Riemannian Manifold Monte Carlo are pretty cutting edge (although are implemented in popular libraries like MC-Stan and Pymc3) and both require fairly advanced physics to really understand.

Additionally, we're seeing the introduction of even more sophisticated stochastic samplers (stochastic gradient hamiltonian monte-carlo, etc) that require even more esoteric branches of math and physics to really grok. I have a strong math background but frequently find myself struggling with a lack of knowledge in statistical mechanics when trying to read papers in these areas.

So yeah - there's plenty of bullshit and exaggeration. But there's also some wicked cool stuff happening which requires very sophisticated (and specialized) knowledge to understand.