Ask HN: What are the foundational texts for learning about AI/ML/NN?

https://books.google.com/books/about/Neural_Smithing.html?id...

Take a look at the Google Books preview (click view sample). The basics are all there, intro to biological history of neural networks, backpropagation, gradient descent, and partial derivatives etc. It even hints at teacher-student methods!

The only issue is that it missed out on two decades of hardware development (and a bag of other optimization tricks). Modern deep learning implementations requires machine sympathy at scale. It also doesn't have any literature on autoregressive networks like RNNs or image processing tricks like CNNs.

Now I think you've got key parts. There's how to use recent production ready models/systems, how to train them and how to make them. Is it in a research or business context?

The field is also broad enough that any one section (text, images, probably symbols) and subsection (time series, bulk, fast online work) all have significant bodies of work behind them. My splits here will not be the best currently so I'm happy for any corrections on a useful hierarchy by the way.

Perhaps you're interested in the history and what's led up to today's work? That's more of a "brief history of time" style coverage, but illuminating.

I'm aware I've not helpfully answered, but I think the same question could have very different valid goals and wanted to bring that to the fore.

I adore PRML, but the scope and depth is overwhelming. LfD encapsulates a number of really core principles in a simple text. The companion course is outstanding and available on EdX.

The tradeoff is that LfD doesn't cover a lot of breath in terms of looking at specific algorithms, but your other texts will do a better job there.

My second recommendation is to read the documentation for Scikit.Learn. It's amazingly instructive and a practical guide to doing ML in practice.

You'd need the following background:

- Linear Algebra

- Multivariate Calculus

- Probability theory && Statistics

Then you need a decent ML book to get the foundations of ML, you can't go wrong with either of these:

- Bishop's Pattern Recognition

- Murphy's Probabilistic ML

- Elements of statistical learning

- Learning from data

You can supplement Murphy's with the advanced book. Elements is a pretty tough book, consider going through "Introduction to statistical learning"[1]. Bishop and Murphy include foundational topics in mathematics.

LfD is a great introductory book and covers one of the most important aspects of ML, that is, model complexity and families of models. It can be supplemented with any of the other books.

I'd also recommend doing some abstract algebra, but it's not a prerequisite.

If you would like a top-down approach, I recommend getting the book "Mathematics of Machine Learning" and learning as needed.

For NN methods, some recommendations:

- https://paperswithcode.com/methods/category/regularization

- https://paperswithcode.com/methods/category/stochastic-optim...

- https://paperswithcode.com/methods/category/attention-mechan...

- https://paperswithcode.com/paper/auto-encoding-variational-b...

For something a little bit different but worth reading given that you have the prerequisite mathematical maturity

- Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges | https://arxiv.org/abs/2104.13478

[1] https://www.statlearning.com/

Many thanks to the user "mindcrime" for catching my error with Introduction to statistical learning.