Ask HN: What are the foundational texts for learning about AI/ML/NN? • Hacker News Board

Re: Ask HN: What are the foundational texts for learning about AI/ML/NN?

#71

Post by PartiallyTyped » Mon, Jan 09, 2023, 9:00 PM UTC

sillysaurusx wrote: post #57

6gvONxR4sf7o wrote: post #53

Earlier quoted context omitted.

I disagree strongly. In your analogy, if the compiler broke down all the time, you would probably need to understand assembly to do programming. ML is amazing today, but still kinda sucks. In general you’ll have a bunch of failures on the way to a successful novel application, so it’s more critical to understand what’s going on under the hood in ML than in your programming analogy. If you just want to apply well know…

We agree, I think! And certainly, if you're one of those people who can pull it off, studying ML from first principles is probably an advantage. I just wince every time since I wouldn't have gotten into ML in the first place if I had to start with a big Calculus tome. There are probably a lot of people like me out there.

OP asked for foundational, and I provided _foundational_. In my opinion, everyone should start from some sound foundations in LinAlg and Calculus.

Here are a couple of errors that stem from a single foundational problem:

- a linear regressor can not be more than the number of datapoints

- dimensionality reduction when you have NxM with M > N is bogus and you need a bigger dataset to do anything meaningful other than clustering

- input dimension of output layer is larger than the number of samples

The underlying issue in all of these is the rank nullity theorem which is pretty foundational for ML, and yet many practitioners don't know about it or haven't made the connection.

I am not expressing that you should have gone through Spivak or build bottom up. There are books like mathematics of ML that condense everything you need, giving you a decent enough foundation for what you will need.

Re: Ask HN: What are the foundational texts for learning about AI/ML/NN?

#72

Post by gerash » Mon, Jan 09, 2023, 9:01 PM UTC

I'd suggest these two by Kevin Murphy:

Probabilistic Machine Learning: An Introduction

https://probml.github.io/pml-book/book1.html

Probabilistic Machine Learning: Advanced Topics

https://probml.github.io/pml-book/book2.html

Re: Ask HN: What are the foundational texts for learning about AI/ML/NN?

#73

Post by davidhunter » Mon, Jan 09, 2023, 9:06 PM UTC

The Quest for Artificial Intelligence: A History of Ideas and Achievements Nils J. Nilsson

This is a good overview of the history of the field (up to SVMs and before deep NNs). I found this useful for putting all the different approaches into context.

Re: Ask HN: What are the foundational texts for learning about AI/ML/NN?

#74

Post by digitalsushi » Mon, Jan 09, 2023, 9:13 PM UTC

Are there obvious paths into these spaces for someone stuck over in devops/infrastructure/platform engineering? Or is it too far a hop to really find a direct path in?

Let me ask a slightly different way - can someone like me get into a job like these, without needing some more college?

My day job is wrapping up OS templates for people with ML software and I always wonder what they get to go do with them once they turn into a compute instance.

Re: Ask HN: What are the foundational texts for learning about AI/ML/NN?

#75

Post by bilsbie » Mon, Jan 09, 2023, 9:15 PM UTC

If anyone is just starting and out wanting to do a study group let me know.

I’m having trouble keeping my motivation up but I really want to get up to speed on how LLM’s work and someday make a career switch.

Re: Ask HN: What are the foundational texts for learning about AI/ML/NN?

#76

Post by PeterisP » Mon, Jan 09, 2023, 9:15 PM UTC

jimbokun wrote: post #66

rg111 wrote: post #62

Earlier quoted context omitted.

You are plain exaggerating. You can't do all of them in a few weeks. Algorithms: Lin Reg -> Log Reg -> NN -> CNN + RNN -> GANs + Transformers -> ViT -> Multimodal AI + LLMs + Diffusion + Auto Encoders SVM, PCA, kNN, k-means clustering, etc. LightGBM, XGboost, Catboost, etc. Optimization and optimizers. Application-wise: Classification, Semantic Segmentation, Pose Estimation, Text Generation, Summarization, NER, Image…

> SVM, PCA, kNN, k-means clustering Are these still relevant in the age of Deep Neural Networks?

Yes, there are all kinds of tasks where the appropriate solution is to use a DNN for much of the learning (either directly learning the correlations or as transfer learning from some large-data self-supervised task) and then, once you have the results of that DNN inference, work with these methods - apply PCA for interpreting the resulting vector, or to separate out specific dimensions to expose them for adjustment in some generative task; or perhaps the best way for the final decision is a kNN on top of the DNN output, etc.

Re: Ask HN: What are the foundational texts for learning about AI/ML/NN?

#77

Post by PartiallyTyped » Mon, Jan 09, 2023, 9:15 PM UTC

mindcrime wrote: post #56

PartiallyTyped wrote: post #19

I recommend against DL by Goodfellow. At this point it is pretty much outdated. Actually, anything specific to NNs is already outdated by release. You'd need the following background: - Linear Algebra - Multivariate Calculus - Probability theory && Statistics Then you need a decent ML book to get the foundations of ML, you can't go wrong with either of these: - Bishop's Pattern Recognition - Murphy's Probabilistic ML…

consider going through "Introductions to Elements of statistical learning" Was that supposed to be An Introduction to Statistical Learning [1] or maybe Introduction to Statistical Relational Learning [2]? I don't think there is a book titled Introduction to Elements of Statistical Learning ? [1]: https://www.statlearning.com/ [2]: https://www.cs.umd.edu/srl-book/

I referred to [1], thanks I have corrected GP.

Re: Ask HN: What are the foundational texts for learning about AI/ML/NN?

#78

Post by friendlyHornet » Mon, Jan 09, 2023, 9:18 PM UTC

digitalsushi wrote: post #74

Are there obvious paths into these spaces for someone stuck over in devops/infrastructure/platform engineering? Or is it too far a hop to really find a direct path in? Let me ask a slightly different way - can someone like me get into a job like these, without needing some more college? My day job is wrapping up OS templates for people with ML software and I always wonder what they get to go do with them once they tu…

I would like to know this, as well.

Re: Ask HN: What are the foundational texts for learning about AI/ML/NN?

#79

Post by jtmcmc » Mon, Jan 09, 2023, 9:22 PM UTC

bjornsing wrote: post #21

softwaredoug wrote: post #2

"Introduction to Statistical Learning" - https://www.statlearning.com/ (there's also "Elements of Statistical Learning" which is a more advanced version) AI: A Modern Approach - https://aima.cs.berkeley.edu/

The Elements of Statistical Learning, by Jerome H. Friedman, Robert Tibshirani, and Trevor Hastie. I’ve seen it referenced quite a few times and the TOC looks good.

This was one of the first books my advisor told me to read when I started my ML phd a...long time ago. The fundamentals of machine learning haven't changed and it's a great book.

Re: Ask HN: What are the foundational texts for learning about AI/ML/NN?

#80

Post by zmgsabst » Mon, Jan 09, 2023, 9:23 PM UTC

digitalsushi wrote: post #74

Are there obvious paths into these spaces for someone stuck over in devops/infrastructure/platform engineering? Or is it too far a hop to really find a direct path in? Let me ask a slightly different way - can someone like me get into a job like these, without needing some more college? My day job is wrapping up OS templates for people with ML software and I always wonder what they get to go do with them once they tu…

Why not ask them?

Call it cross functional training to increase your domain knowledge, tell your manager you need it to ensure you’re providing the best service possible, and get your coworkers to help you learn the framework they use…?