Stay away, in my opinion. I spent a year supporting a SVM in a production machine learning application, and it made me wish the ML research community hadn't been so in love with them for so long. They're the perfect blend of theoretically elegant and practically impractical. Training scales as O(n^3), serialized models are heavyweight, prediction is slow. They're like Gaussian Processes, except warped and without any…
I’d agree on the training time but your serialized model should be small on disk since only the support vectors are needed for inference. At least with my experience that has been true.
If you're new to ML or datascience, I would recommend working to build a strong basis in Bayesian statistics. It will help you understand how all of the "canonical" ML methods relate to one another, and will give you a basis for building off of them. In particular, aspire to learn probabilistic graphical models + the libraries to train them (like pyro, tensorflow probability, Edward, Stan). They have a steep learning…
Thanks for this insight. Can you kindly also suggest a good book for someone to start with Bayesian Statistics? I could really use a suggestion about first and second book on this. About Probabilistic Graphical Models, is there book other than Daphne Koller's book that you would suggest?
Thanks, everyone, for your kind suggestions. Much appreciated.
Starting with PGMs would kill 99.9% of aspiring ML practitioners. Classes related to PGM at Stanford and MIT are considered to be some of the most difficult ones. I'd rather recommend to start with something they are enthusiastic about and once they become sufficiently advanced, to naturally learn (H)PGM.
I think they said "aspire to", not start with. I would start with a naive bayes classifier, understand why it is naive and then go from there.
Exactly. I’m talking about how to orient yourself, PGMs are hard, sure but they don’t have to be outrageously hard. I would argue that if you do understand naive bayes (and it’s naivety) and you understand priors, then PGMs are just like rules to a game called “make a diagram of your posterior”. That’s not all there is obviously, and that’s kind of my point; you can do a lot with a little bit of knowledge, and then you can slowly climb that ladder for a long time and the more you learn, the more you can apply. Starting with an ad hoc approach (here are all of these classes in scikit learn with .fit() functions, lets just memorize their docstrings) isn’t “bad” per se, that knowledge is important, but it will not take you very deep, and you won’t be able to stray very far from those methods without being out of your depth and running into trouble.
I've been an ML practioner since 2009. I've used every method imaginable or popular, I think. With the exception of non-linear SVMs. Linear SVM => All good, just the hingle loss optimization. Non-linear SVM, a bit of overkill with basis expansion. Just too slow, or too complex a model?
My impression: SVMs are more of theoretical interest than practical interest. Yeah, learn your statistics. Loss functions. Additive models. Neural nets. Linear models. Decision trees, kNNs etc. SVM is more of a special interest, imho.
If you're new to ML or datascience, I would recommend working to build a strong basis in Bayesian statistics. It will help you understand how all of the "canonical" ML methods relate to one another, and will give you a basis for building off of them. In particular, aspire to learn probabilistic graphical models + the libraries to train them (like pyro, tensorflow probability, Edward, Stan). They have a steep learning…
Different people learn in different ways, but personally I’ve had more success with the opposite approach, ie “top-down”. As in, rather than learning in depth all the low level parts then finally putting it together at the end, start with a surface high-level understanding of a working prototype then expand into the details of how everything works inside. In the case of ML, this could mean starting with a 5 line SciK…
I support this learning method. Having a map of the key concepts, as well as visceral feel for them via code, will keep up the motivation.
This would be in contrast to picking up one of the plethora of “ML” textbooks that mostly only describe the math behind all the algorithms. Which is not where you should begin, in my view (years of teaching experience). The use of such textbooks is as a reference to fill in details once your are curious about them.
And more than anything, the best way to learn practical ML is to “apprentice” to some experienced practitioners or team who are willing to act as mentors.
If you're new to ML or datascience, I would recommend working to build a strong basis in Bayesian statistics. It will help you understand how all of the "canonical" ML methods relate to one another, and will give you a basis for building off of them. In particular, aspire to learn probabilistic graphical models + the libraries to train them (like pyro, tensorflow probability, Edward, Stan). They have a steep learning…
Different people learn in different ways, but personally I’ve had more success with the opposite approach, ie “top-down”. As in, rather than learning in depth all the low level parts then finally putting it together at the end, start with a surface high-level understanding of a working prototype then expand into the details of how everything works inside. In the case of ML, this could mean starting with a 5 line SciK…
Agree — people should do what they want and have fun learning. Its just a suggestion that’s also colored by my own experience. I will argue that if you’re going for a job in data science it is a bit more of a serious suggestion since you need to be able to know how to answer statistical questions and understand your assumptions, and you really do need to understand Bayesian stats for that (nothing state of the art here, don’t worry if you don’t know what a PGM is, I just mean basics).
Thanks for this insight. Can you kindly also suggest a good book for someone to start with Bayesian Statistics? I could really use a suggestion about first and second book on this. About Probabilistic Graphical Models, is there book other than Daphne Koller's book that you would suggest?
Introduction to Statistical Learning https://faculty.marshall.usc.edu/gareth-james/ISL/ Elements of Statistical Learning https://web.stanford.edu/~hastie/ElemStatLearn/ Machine Learning: A Probabilistic Perspective https://mitpress.mit.edu/books/machine-learning-1
Are those really the best starts for "Bayesian statistics"?
Especially the first 2 are rather the standard "intro to ML textbooks", with a frequentist focus (ISL may even have zero Bayesian stuff - Naive Bayes is not "Bayesian" – while ESL still has maybe 10% bayesian content if that).
If you're new to ML or datascience, I would recommend working to build a strong basis in Bayesian statistics. It will help you understand how all of the "canonical" ML methods relate to one another, and will give you a basis for building off of them. In particular, aspire to learn probabilistic graphical models + the libraries to train them (like pyro, tensorflow probability, Edward, Stan). They have a steep learning…
Personally I'd advise against both SVM's and Bayesian methods for a beginner. Bayesian statistics is very much the deep end of the pool. Graphical models and Bayesian methods generally may make a comeback but such approaches have been superseded by other methods for good reasons, i.e. scaling. A strong basis in statistics is certainly a great thing, but that can be maximum likelihood plus Bayes law (i.e. "MAP" estima…
>The training algorithms for deep learning are also the hottest algorithm research area in machine learning, and are certainly applicable beyond deep learning.
The lore I've heard is that most new deep learning training algorithms (optimization algorithms) only work better on particular special cases, and it is hard to do better than the established algorithms in general.
I'm also not sure why you're saying they're applicable beyond deep learning--how do you plan to train a PGM or SVM using Adam?
Can you give a concrete example how a Bayesian concept provides practical help in ML model training or another ML task? Want to whet my appetite for your suggestion.
A common type of example involves relatively small or uninformative datasets. Say you flip a coin a few times and only get heads. Your maximum likelihood (frequentist) estimate is that the coin will always land heads. In a Bayesian setting, if you have a (say uniform) prior on the probability that the coin lands heads, your maximum a posteriori estimate of this probability will be non-zero, but will get continue to g…
>Say you flip a coin a few times and only get heads. Your maximum likelihood (frequentist) estimate is that the coin will always land heads. In a Bayesian setting, if you have a (say uniform) prior on the probability that the coin lands heads, your maximum a posteriori estimate of this probability will be non-zero, but will get continue to get smaller if you continue only seeing heads.
Not quite. If you have a uniform prior, there will be no difference between MAP and MLE.
>From the vantage point of Bayesian inference, MLE is a special case of maximum a posteriori estimation (MAP) that assumes a uniform prior distribution of the parameters.
Introduction to Statistical Learning https://faculty.marshall.usc.edu/gareth-james/ISL/ Elements of Statistical Learning https://web.stanford.edu/~hastie/ElemStatLearn/ Machine Learning: A Probabilistic Perspective https://mitpress.mit.edu/books/machine-learning-1
Are those really the best starts for "Bayesian statistics"? Especially the first 2 are rather the standard "intro to ML textbooks", with a frequentist focus (ISL may even have zero Bayesian stuff - Naive Bayes is not "Bayesian" – while ESL still has maybe 10% bayesian content if that). Instead, I would suggest the following for learning Bayesian methods, especially given the HN crowd: https://github.com/CamDavidsonPi…
You make a good point. It's been a while since I flipped through them, they just come up in lots of discussions on this topic. I agree that the series you link to is really great for PPL and Bayesian methods. You may find that the library upon which it's based (PyMC3) is built on top of Theano, which has been abandoned and deprecated. PyMC4 is around the corner and uses TensorFlow Probability. Early, informal reports say it's 10x faster.