An Introduction to Statistical Learning with Applications in Python

People like you are funny because you don't realize I'm actually also calling out a lot of ivory tower people.

Also, I'm not saying you need the math to train a model. I'm not sure you even need linear algebra to do that, mostly just programming. That's why I called it the beginning. I was directly responding to your claim that this is all you need __to understand__. Because let's be real, you don't need to know backprop (and thus derivatives and chain rule) to train models. The math is about how to analyze your models. You know, specifically what academia is supposed to be doing. Research and engineering overlap but they aren't necessarily the same thing.

Besides, math education is notorious for being essentially free. Who needs Stanford/Berkeley/MIT when textbooks exist widely. (Btw, CS doesn't typically produce mathematicians)

It's also a LOT more explainable and interpretable.

>and might mean you can guess at the roots of mine :)

your services are required on the busy beaver thread!

>It is really impressive that people who are working with probability and likelihood every day do not understand the difference

i think i come away from the whole experience (the phd, even though i'm not done yet) with a deep skepticism/cynicism of very many things. but it's probably not what you expect. i just don't think the math is at all relevant/important epistemically as long as you can run the experiments efficiently. which is exactly what you see happening - people with access to gobs of compute develop good intuition that leads them towards breakthroughs, and people that don't have access to compute struggle and make do with the formalisms. it's not much different in physics, where the good experimentalists aren't born that way, they're made in the well-funded labs.

i firmly believe that in ML, the math does not matter at all, beyond the tiny bit of calculus and linear algebra you need to kind of understand forwards and backwards. of course everytime i say this on here i'm skewered/debated to death on it, as if i don't know what i'm talking about :shrug:

[1] CMU 10-701: Introduction to Machine Learning:

    Recommended Textbooks:
       
       Pattern Recognition and Machine Learning, Christopher Bishop
       
       Machine Learning: A probabilistic perspective, Kevin Murphy

[2] University of Toronto CSC 311: Introduction to Machine Learning

    Suggested readings are optional; they are resources we recommend to help you understand the course material. All of the textbooks listed below are freely available online.
    
      Bishop = Pattern Recognition and Machine Learning, by Chris Bishop
    
      ESL = The Elements of Statistical Learning, by Hastie, Tibshirani, and Friedman. 

[3] EPFL CS-433 Machine Learning:

    Textbooks(not mandatory)

      Gilbert Strang, Linear Algebra and Learning from Data
    
      Christopher Bishop, Pattern Recognition and Machine Learning

[4] University of Washington CSE 446: Machine Learning

    The required textbook for the course is:

      [Murphy] Machine Learning: A Probabilistic Perspective, Kevin Murphy.

    The following three texts are also excellent and their PDFs are available for free online.

      [B] Pattern Recognition and Machine Learning, Christopher Bishop.

      [HTF] The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Trevor Hastie, Robert Tibshirani, Jerome Friedman.

[5] Cornell University ECE4950: Machine Learning and Pattern Recognition

    Materials

    We will take materials from various sources. Some books are:

      Pattern Recognition and Machine Learning, Christopher Bishop

      Machine Learning: a Probabilistic Perspective, Kevin Murphy

[6] Princeton University COS 324: Introduction to Machine Learning

    Optional Machine Learning Books

      [Murphy] Kevin Murphy, Machine Learning: A Probabilistic Perspective, MIT Press.

      [Bishop] Christopher M. Bishop, Pattern Recognition and Machine Learning, Springer.

[7] ETH Zurich Introduction to Machine Learning (2023)

    Other Resources

      K. Murphy. Machine Learning: a Probabilistic Perspective. MIT Press, 2012.

      C. Bishop. Pattern Recognition and Machine Learning. Springer, 2007.

[8] TUM (Technical University of Munich) Machine Learning

    This award-winning introductory Machine Learning lecture teaches the foundations of and concepts behind a wide range of common machine learning models.

    Literature

      Pattern Recognition and Machine Learning. Christopher Bishop. Springer-Verlag New York. 2006.

      Machine Learning: A Probabilistic Perspective. Kevin Murphy. MIT Press. 2012

[9] MIT Introduction To Machine Learning:

    Books: No textbook is required for this class, but students may find it helpful to purchase one of the following books. Bishop's book is much easier to read, whereas Murphy's book has substantially more depth and coverage (and is up to date).

      Machine Learning: a Probabilistic Perspective, by Kevin Murphy (2012).

      Pattern Recognition and Machine Learning, by Chris Bishop (2006).

[10] UC Berkeley CS-194-10: Introduction to Machine Learning:

    Reading List (Preliminary Draft)

    The first two books are very helpful, and are available online, so those (in addition to AIMA) will be the primary sources. Bishop has a wide range of solid mathematical derivations, while Witten and Frank focus much more on the practical side of applied machine learning and on the Weka package (a Java library and interface for machine learning).

      Trevor Hastie, Rob Tibshirani, and Jerry Friedman, Elements of Statistical Learning, Second Edition, Springer, 2009. (Full pdf available for download.)

      Kevin P. Murphy, Machine Learning: A Probabilistic Perspective. Unpublished. Access information will be provided.

      Stuart Russell and Peter Norvig, Artificial Intelligence: A Modern Approach, Third Edition, Prentice Hall, 2010.

      Christopher Bishop, Pattern Recognition and Machine Learning, Springer, 2006.

      Ian Witten and Eibe Frank, Data Mining: Practical Machine Learning Tools and Techniques, Third Edition, Morgan Kaufmann, 2011. 

[1] https://www.cs.cmu.edu/~aarti/Class/10701_Spring21/

[2] https://www.cs.toronto.edu/~rgrosse/courses/csc311_f21/

[3] https://www.epfl.ch/labs/mlo/machine-learning-cs-433/

[4] https://courses.cs.washington.edu/courses/cse446/23wi/textbo...

[5] https://people.ece.cornell.edu/acharya/teaching/ece4950s17/e...

[6] https://www.cs.princeton.edu/courses/archive/spring19/cos324...

[7] https://las.inf.ethz.ch/teaching/introml-s23

[8] https://www.cs.cit.tum.de/en/daml/lehre/wintersemester-2022-...

[9] https://people.csail.mit.edu/dsontag/courses/ml16/

[10] https://people.eecs.berkeley.edu/~russell/classes/cs194/f11/

> your services are required on the busy beaver thread!

Lol I didn't even see it. I'm assuming this is w.r.t Mutual Information's video?

> the phd, even though i'm not done yet

I'm at about a similar point (last year). Most of my cynicism though is around academia and publishing. Making conferences the de facto target for publishing was a mistake. Zero-shot submissions in a zero-sum game environment? Can't see how that would go wrong...

> it's not much different in physics, where the good experimentalists aren't born that way, they're made in the well-funded labs.

Coming from the experimental physics side (my undergrad), there is a big factor though. Generally the experimentalists who were good at the math and could learn to intuit them (and especially the uncertainty) did better. But you're absolutely correct about the __well funded__ part being a big indicator. When I've worked at gov labs I didn't notice a quality difference in intellect between peers from different schools (of a wide variety of prestige) but what did stand out was simply experience. Your no-name school physicists could pick up the skills fast, but they just never had opportunities like the prestigious school students did. It didn't make too big of a difference, but it is an interesting note, especially since it tells us how to make more of those higher status researchers...

> i firmly believe that in ML, the math does not matter at all

My opinion is that this is highly context dependent. Most research right now is about optimization and tuning, and with respect to that, I fully agree. I'm including in that even some architecture search, such as "replace CNN with Transformer" and such things. This you can do pretty much empirically. The only big point I'll get on here is that people do not understand the limitations of their metrics (especially parametric metrics), biases of the datasets, and the biases of their architectures, so it creates a really weird environment where we aren't comparing things fairly. (It is also why what works in research doesn't always work out well in industry) But if we're talking about interpretability, understanding, novel architecture design, evaluation methods, and so on, then I do think it matters. There's a lot that we can actually understand about ML -- how they work and how they form answers -- that isn't discussed not because it hasn't been researched but because the research has a higher barrier to entry and people don't even understand the results. It isn't uncommon to see a top tier paper empirically find what a theoretical paper (with experiments, but lower compute) found 5-10 years back, where the recent work didn't even know about the prior work. Where the higher level math really helps out is being able to read deeper and evaluate deeper. Fwiw, every time I make a "math is necessary" argument, I get a lot of people pushing back. But I think this is because both groups have two camps. For the pro math I believe there is people who legitimately believe it (like me) -- who usually talk about high dimensional statistics and other things well past calculus -- and people who say it to make themselves feel smart -- people who often think calculus is high level math or say "linear algebra" as if it is just what's in David Lay's book. For the anti-math crowd I think there are the hype people who just don't care and the people who are just doing other things and don't really end up using it. For the latter, I do think they are still benefiting a lot from the intuition about these systems that they gained from those math courses. But then again, the classic thing in math education is that you struggle while you learn it and then after you know it it is trivial.

For research, I firmly believe you need both the high math and the "knob turners." I just think academia and conferencing should be focused around the former and industry should focus around the latter. But the problem is we have these people operating in the exact same space and we're comparing works that use 100+yrs of compute hours to works that have a month or two of compute hours. This isn't a great way to really tell if one architecture is better than another since hyperparameters matter so much. It's just making for bad research and railroading.