Learning About Statistical Learning

"Mathematical Statistics and Data Analysis" by John A. Rice

"All of Statistics: A Concise Course in Statistics" by Larry Wasserman

"Pattern Recognition and Machine Learning" by Christopher M. Bishop

"The Elements of Statistical Learning" by T. Hastie et al http://www-stat.stanford.edu/~tibs/ElemStatLearn/

"Information Theory, Inference, and Learning Algorithms", David McKay http://www.inference.phy.cam.ac.uk/itprnn/book.html

"Introduction to Information Retrieval" - Manning et al. http://nlp.stanford.edu/IR-book/information-retrieval-book.h...

"The Algorithm Design Manual, 2nd Edition" - Steven Skiena http://www.algorist.com/

I have two points.

First, your point about programming is incredibly important. I've worked with people who had amazing insights about statistical problems, but went cross-eyed upon being asked about SVN and Git. This makes a CS homework assignment unpleasant, and a real world research project impossible.

Second, this really begs another post. It should be called "Learning how to read a textbook on your own." Successful self-learners don't just __read__ a textbook. They toil with it, try proving things on paper themselves, work through exercises, attempt to apply it to some real-world situation, and hunt down someone who's smarter than they are to explain something that seems unclear.

Not all textbooks - certainly not every one on your list - can be read with a great application in mind. A reader must interrogate mercilessly the book on analysis or the rigorous probability book mentioned in the post.

This seems intuitive to someone who can successfully learn on their own, but most people are not taught to do that. The difficulties of relicating a portion of the classroom learning experience is a major barrier to entry. This is why online intro lectures for programming, math, and certain CS topics like algorithms can steer a learner in the right direction. Stanford Engineering Everywhere and of course MIT's OCW, links to which have been posted on HN at least once a month, are great starts.

I hope someone finds this useful.

For those who want a summary,

Proof Technique

(a)Velleman's "How to Prove It" (b)Gries and Schneider's "A Logical Approach to Discrete Math"

Math

(c) Calculus (best "lite" book - Calculus by Strang (free download), best "heavy" books - (d) Calculus by Spivak, (e) Principles of Mathematical Analysis a.k.a "Baby Rudin")

(f) Discrete Math (ALADM above + (g) a good book on Algorithms, Cormen will do - though working through it comprehensively is ... hard!

(h) Linear Algebra (First work through Strang's book, then (i) Axler's)

(j) Probability (see Bradford's very comprehensive recommendations) and

(k) Statistics (I would reccomend Devore and Peck for the total beginner but it is a damn expensive book. So hit a library or get a bootlegged copy to see if it suits you before buying a copy, see brad's list for advanced stuff.)

(l) Information Theory (MacKay's book is freely available online)

Basic AI

(m)AIMA 3d Edition (I prefer this to Mitchell)

Machine Learning

(n) "Pattern Recognition and Machine Learning" by Christopher Bishop,

(o)"Elements of Statistical Learning" (free download).

(p) Neural Network Design by Hagan Demuth and Kneale,

(q) Neural Networks, A Comprehensive Foundation (2nd edition) - By Haykin (there is a newer edition out but I don't know anything about that, this is the one I used)

(r) Neural Networks for Pattern Recognition ( Bishop).

At this point you are in good shape to read any papers in NN. My reccomendations - anything by Yann LeCun and Geoffrey Hinton. Both do amazing research.

Reinforcement Learning

(s) Reinforcement Learning - An Introduction by Barto and Sutton (follow up with "Recent Advances In reinforcement Learning" (PDF) which is an old paper but a GREAT introduction to Hierarchical Reinforcement learning)

(t) Neuro Dynamic Programming by Bertsekas

Computer Vision

(u) Introductory Techniques for 3-D Computer Vision, by Emanuele Trucco and Alessandro Verri.

(v) An Invitation to 3-D Vision by Y. Ma, S. Soatto, J. Kosecka, S.S. Sastry. (warning TOUGH!!)

Robotics.

(w) Probabilistic Graphical Models: Principles and Techniques (Adaptive Computation and Machine Learning) - not about robotics per se but useful to understand the next book

(x) Probabilistic Robotics (Intelligent Robotics and Autonomous Agents) by Thrun, Burgard and Fox

PS: I own all these books (except AIMA 3 for which I only have pre pub pdfs) and if any HN folks in Bangalore want to browse before they buy anything (friggin expensive when you add amazon's postage) send me email.

PPS: on languages, I think Bradford is on the money with regards to reccomending functional languages. I would just say, also know C well. Saved my ass a few times.

At a minimum, I would recommend learning python (numpy/scipy), R, and at least one nice functional language (probably Haskell, Clojure, or OCaml).

This is effectively what I use as well. Python as a general purpose data munging library that's good for all of your dirty work whenever you need it. R for graphing, graphing, graphing, running statistical tests other people already wrote and foolproofed, database munging, and then more graphing. Haskell for prototyping and reasoning with types and then that occasional algorithm that screams for functional implementation or the not so occasional one that requires more speed than Python can provide.

I also write a few things in C/C++, though I try to avoid it. It's mostly there for standing on the backs of other people and that occasional need to blaze.