I maintain a list of well-known or foundational papers in ML in a github repo that may be of interest to readers of this thread
Ask HN: What are the foundational texts for learning about AI/ML/NN?
61–70 of 114 posts
Re: Ask HN: What are the foundational texts for learning about AI/ML/NN?
#62There are none anymore. We now know that throwing a bunch of bits into the linear algebra meat grinder gets you endless high quality art and decent linguistic functionality. The architecture of these systems takes maybe a week to deeply understand, or maybe a month for a beginner. That's really it. Everything else is obsolete or no longer applicable unless you're interested in theoretical research on alternatives to…
You are plain exaggerating. You can't do all of them in a few weeks.
Algorithms:
Lin Reg -> Log Reg -> NN -> CNN + RNN -> GANs + Transformers -> ViT -> Multimodal AI + LLMs + Diffusion + Auto Encoders
SVM, PCA, kNN, k-means clustering, etc.
LightGBM, XGboost, Catboost, etc.
Optimization and optimizers.
Application-wise:
Classification, Semantic Segmentation, Pose Estimation, Text Generation, Summarization, NER, Image Generation, Captioning, Sequence Generation (like music/speech), text to speech, speech to text, recommender systems, sentiment amalysis, tabular data, etc.
Frameworks:
pandas, sklearn, PyTorch, Jax -> training inference, data loading
Platforms:
AWS + GCP + Azure
And a lot of GPU shenanigans + framework/platform specific quirks
given that:
- you already know Python/any programming language properly
- you already know college level math (many people say you don't need it, but haven't met a single soul in ML research/modelling without college level math)
- you know Stats 101 matching a good uni curriculum and ability to learn beyond
- you know git, docker, cli, etc.
Every influencer and their mother promising to teach you Data Science in 30 days are plain lying.
Edit: I see that I left out Deep RL. Let's keep it that way for now.
Edit2: Added tree based methods. These are very important. XGBoost outperforms NNs every time on tabular data. I also once used an RF head appended to a DNN, for final prediction. Added optimizers.
Re: Ask HN: What are the foundational texts for learning about AI/ML/NN?
#63I personally consider Linear algebra to be foundational in AI/ML. Intro to Linear algebra, Gilbert Strang. And his free course on MIT OCW is fantastic too. While having strong mathematical foundation is useful, I think developing intuition is even more important. For this, I recommend Andrew Ng's coursera courses first before you dive too deep.
Another interesting resource for Linear Algebra is the "Coding the Matrix" course.
https://www.youtube.com/playlist?list=PLEhMEyM9jSinRHXJgRCOL...
Re: Ask HN: What are the foundational texts for learning about AI/ML/NN?
#64- AIMA by Russel and Norvig is a classic but I would say is more of overview of the field and for most topic areas isn't quite deep enough imo. - For deep learning specifically, a more applied text that is beautifully written and chock full of examples is Francois Chollet's Deep Learning with Python (there a new second edition out with up to date examples using modern versions of Tensorflow). The first 3 chapters I w…
Norvig-Russel has many chapters spanning hundreds of pages that are way out of date and not used anywhere.
And the new things he cover are covered in a better manner and better depth in other sources.
I read this book like a novel. Good for a basic overview, but the RoI is very low.
Re: Ask HN: What are the foundational texts for learning about AI/ML/NN?
#65There are none anymore. We now know that throwing a bunch of bits into the linear algebra meat grinder gets you endless high quality art and decent linguistic functionality. The architecture of these systems takes maybe a week to deeply understand, or maybe a month for a beginner. That's really it. Everything else is obsolete or no longer applicable unless you're interested in theoretical research on alternatives to…
A month to deeply understand? I've been doing it since early 2019 and there are still subtleties that catch me off guard. Get back to me when you're not surprised that you can get rid of biases from many layers without harming training. I broadly agree with you, but the timeline was just a little too aggressive. By about 10x. :)
This is separate from understanding how a language model or transformer works. You could read the major papers behind those ideas and read every line of code involved several times over in a month. I'd recommend it, if you're super curious.
You can figure out the bias thing after about a month (or so) of hands on practice. Do one Kaggle seriously and it'll become pretty clear, pretty quickly.
Re: Ask HN: What are the foundational texts for learning about AI/ML/NN?
#66There are none anymore. We now know that throwing a bunch of bits into the linear algebra meat grinder gets you endless high quality art and decent linguistic functionality. The architecture of these systems takes maybe a week to deeply understand, or maybe a month for a beginner. That's really it. Everything else is obsolete or no longer applicable unless you're interested in theoretical research on alternatives to…
You are plain exaggerating. You can't do all of them in a few weeks. Algorithms: Lin Reg -> Log Reg -> NN -> CNN + RNN -> GANs + Transformers -> ViT -> Multimodal AI + LLMs + Diffusion + Auto Encoders SVM, PCA, kNN, k-means clustering, etc. LightGBM, XGboost, Catboost, etc. Optimization and optimizers. Application-wise: Classification, Semantic Segmentation, Pose Estimation, Text Generation, Summarization, NER, Image…
> SVM, PCA, kNN, k-means clustering
Are these still relevant in the age of Deep Neural Networks?
Re: Ask HN: What are the foundational texts for learning about AI/ML/NN?
#67Earlier quoted context omitted.
You are plain exaggerating. You can't do all of them in a few weeks. Algorithms: Lin Reg -> Log Reg -> NN -> CNN + RNN -> GANs + Transformers -> ViT -> Multimodal AI + LLMs + Diffusion + Auto Encoders SVM, PCA, kNN, k-means clustering, etc. LightGBM, XGboost, Catboost, etc. Optimization and optimizers. Application-wise: Classification, Semantic Segmentation, Pose Estimation, Text Generation, Summarization, NER, Image…
> SVM, PCA, kNN, k-means clustering Are these still relevant in the age of Deep Neural Networks?
Yes.
Different problems require different solutions.
Sometimes, an NN would be overkill.
And stakeholders in many situations would like insights why the prediction is what it is. NNs are miles behind LogReg in terms of interpretablity.
Re: Ask HN: What are the foundational texts for learning about AI/ML/NN?
#68Earlier quoted context omitted.
Many of the suggestions so far are assuming you have taken undergraduate linear algebra and calculus. I'd start with those two subjects, you really can't build a foundational understanding of modern AI techniques without them.
i did linear algebra and calculus using strang and spivak textbooks. Those were classes i enjoy the most. But most of that stuff has atrophied from my brain over the years, do you recommend redoing those courses fast or can i learn when i need it on demand basis.
Review on an on demand basis.
The main concepts are matrix multiplication and derivatives and their significance. Then you can dig into the specifics and review or expand your knowledge as needed.
Re: Ask HN: What are the foundational texts for learning about AI/ML/NN?
#69Do you have Linear Algebra knowledge, and Stats 101 knowledge?
Then start with ISLR.
Then go and watch Andrew Ng Machine Learning course on Coursera (a new version was added in 2022 that uses Python).
Then read the sklearn book from its maintainers/core devs. It's from O'Reilly.
Then go do the Deep Learning Specialization from deeplearning.ai.
Then do fast.ai course.
If interested in Deep RL, watch David Silver lectures, then read Deep RL in Action by Zai, Brown. Then do the HF course on Deep RL.
This is how you get started. Choose your books based on your personality, needs, and contents covered.
And among MOOCs, I highly suggest the one by Canziani, LeCun from NYU. (I loved the 2020 version.)
The one taught by Fei Fei Li and Andrej Karpathy is nice.
These two MOOCs can substitute classic books based on quality.
I have never read cover to cover any of the famous books. I read a lot from them sticking to specific subjects.
Get to reading papers, finding implementations. Ng + ISLR will give you good grounds. Fast.ai + deeplearning.ai will give you capability to solve real problems. NYU + Tubingen + Stanford + UMich (Justin Johnson) courses will bring you to the edge.
You need a lot of practical experience that aren’t taught anywhere. So, get your hands dirty early. Learn to use frameworks, cloud platforms, etc.
Then start reading papers.
A crystal clear grasp on Math foundations is a must. Get it if you don't have already.
Re: Ask HN: What are the foundational texts for learning about AI/ML/NN?
#70Earlier quoted context omitted.
You are plain exaggerating. You can't do all of them in a few weeks. Algorithms: Lin Reg -> Log Reg -> NN -> CNN + RNN -> GANs + Transformers -> ViT -> Multimodal AI + LLMs + Diffusion + Auto Encoders SVM, PCA, kNN, k-means clustering, etc. LightGBM, XGboost, Catboost, etc. Optimization and optimizers. Application-wise: Classification, Semantic Segmentation, Pose Estimation, Text Generation, Summarization, NER, Image…
> SVM, PCA, kNN, k-means clustering Are these still relevant in the age of Deep Neural Networks?
PCA is a foundational dimension reduction technique, and kNN can be used in conjunction with embeddings.
k-means is still great when you have prior/domain knowledge about the number of groups.