Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introduction to Support Vector Machines in Machine Learning

ranvir.xyz

Post reply on HN
Print view View original

21–30 of 90 posts

Re: Introduction to Support Vector Machines in Machine Learning

#21

Post by _pastel »

smbrian wrote: post #4

Stay away, in my opinion. I spent a year supporting a SVM in a production machine learning application, and it made me wish the ML research community hadn't been so in love with them for so long. They're the perfect blend of theoretically elegant and practically impractical. Training scales as O(n^3), serialized models are heavyweight, prediction is slow. They're like Gaussian Processes, except warped and without any…

100% agree. What's the use case for SVMs?

Linear models are simpler. GBMs are more powerful, more flexible, and faster.

Every ML course I took had 3 weeks of problem sets on VC dimension and convex quadratic optimization in Lagrangian dual-space, while decision tree ensembles were lucky to get a mention. Meanwhile GBMs continue to win almost all the competitions where neural nets don't dominate.

I suspect my professors just preferred the nice theoretical motivation and fancy math.

Re: Introduction to Support Vector Machines in Machine Learning

#22

Post by rangerranvir »

uoaei wrote: post #17
astrophysician wrote: post #12

If you're new to ML or datascience, I would recommend working to build a strong basis in Bayesian statistics. It will help you understand how all of the "canonical" ML methods relate to one another, and will give you a basis for building off of them. In particular, aspire to learn probabilistic graphical models + the libraries to train them (like pyro, tensorflow probability, Edward, Stan). They have a steep learning…

Great comments. I heartily agree and support the statement about probabilistic graphical models. Just to add a couple more facets to this perspective: 'State of the art' does not always mean 'best for your task', and in fact lately depending on your field SOTA sometimes simply means 'unaffordable' for anyone whose budget is under 1 million dollars. Try linear methods first. Ensembles of decent models are usually good…

Great Points. Really appreciated. Will have to put extra effort to learn about the feature engineering part of the problem.

Also, if you know a few things about the data it becomes a little easier to explain what your model is doing and why it is producing those results.

Found a good resource which explained the trust component: https://arxiv.org/pdf/1602.04938.pdf

Re: Introduction to Support Vector Machines in Machine Learning

#23

Post by rangerranvir »

uoaei wrote: post #18
tesseract2 wrote: post #14

Earlier quoted context omitted.

Thanks for this insight. Can you kindly also suggest a good book for someone to start with Bayesian Statistics? I could really use a suggestion about first and second book on this. About Probabilistic Graphical Models, is there book other than Daphne Koller's book that you would suggest?

Introduction to Statistical Learning https://faculty.marshall.usc.edu/gareth-james/ISL/ Elements of Statistical Learning https://web.stanford.edu/~hastie/ElemStatLearn/ Machine Learning: A Probabilistic Perspective https://mitpress.mit.edu/books/machine-learning-1

Thanks a ton for these. Added this to things I know that I don't know list. ;)

Re: Introduction to Support Vector Machines in Machine Learning

#24

Post by kmundnic »

uoaei wrote: post #18
tesseract2 wrote: post #14

Earlier quoted context omitted.

Thanks for this insight. Can you kindly also suggest a good book for someone to start with Bayesian Statistics? I could really use a suggestion about first and second book on this. About Probabilistic Graphical Models, is there book other than Daphne Koller's book that you would suggest?

Introduction to Statistical Learning https://faculty.marshall.usc.edu/gareth-james/ISL/ Elements of Statistical Learning https://web.stanford.edu/~hastie/ElemStatLearn/ Machine Learning: A Probabilistic Perspective https://mitpress.mit.edu/books/machine-learning-1

"Machine Learning: a Probabilistic Perspective" is more an encyclopedia of algorithms I would say, and it has lots of typos. I personally would not recommend it (except for the amount of algorithms that it covers, many of which are usually not found in other books).

Re: Introduction to Support Vector Machines in Machine Learning

#25

Post by antipaul »

astrophysician wrote: post #12

If you're new to ML or datascience, I would recommend working to build a strong basis in Bayesian statistics. It will help you understand how all of the "canonical" ML methods relate to one another, and will give you a basis for building off of them. In particular, aspire to learn probabilistic graphical models + the libraries to train them (like pyro, tensorflow probability, Edward, Stan). They have a steep learning…

Can you give a concrete example how a Bayesian concept provides practical help in ML model training or another ML task?

Want to whet my appetite for your suggestion.

Re: Introduction to Support Vector Machines in Machine Learning

#26

Post by MaxBarraclough »

smbrian wrote: post #4

Stay away, in my opinion. I spent a year supporting a SVM in a production machine learning application, and it made me wish the ML research community hadn't been so in love with them for so long. They're the perfect blend of theoretically elegant and practically impractical. Training scales as O(n^3), serialized models are heavyweight, prediction is slow. They're like Gaussian Processes, except warped and without any…

So you're saying to stay away from SVMs, rather than to stay away from this particular tutorial?

Re: Introduction to Support Vector Machines in Machine Learning

#27

Post by rjvir »

astrophysician wrote: post #12

If you're new to ML or datascience, I would recommend working to build a strong basis in Bayesian statistics. It will help you understand how all of the "canonical" ML methods relate to one another, and will give you a basis for building off of them. In particular, aspire to learn probabilistic graphical models + the libraries to train them (like pyro, tensorflow probability, Edward, Stan). They have a steep learning…

Different people learn in different ways, but personally I’ve had more success with the opposite approach, ie “top-down”.

As in, rather than learning in depth all the low level parts then finally putting it together at the end, start with a surface high-level understanding of a working prototype then expand into the details of how everything works inside.

In the case of ML, this could mean starting with a 5 line SciKit-learn prototype of a random forest model, seeing some working predictions, then expanding knowledge from there - what data is going in and what is coming out? What’s a classifier? What’s a decision tree? Etc

Re: Introduction to Support Vector Machines in Machine Learning

#28

Post by bencw »

antipaul wrote: post #25
astrophysician wrote: post #12

If you're new to ML or datascience, I would recommend working to build a strong basis in Bayesian statistics. It will help you understand how all of the "canonical" ML methods relate to one another, and will give you a basis for building off of them. In particular, aspire to learn probabilistic graphical models + the libraries to train them (like pyro, tensorflow probability, Edward, Stan). They have a steep learning…

Can you give a concrete example how a Bayesian concept provides practical help in ML model training or another ML task? Want to whet my appetite for your suggestion.

A common type of example involves relatively small or uninformative datasets. Say you flip a coin a few times and only get heads. Your maximum likelihood (frequentist) estimate is that the coin will always land heads. In a Bayesian setting, if you have a (say uniform) prior on the probability that the coin lands heads, your maximum a posteriori estimate of this probability will be non-zero, but will get continue to get smaller if you continue only seeing heads.

The above example is contrived, but makes more sense in the case of language modelling. Since a bag-of-words vector, containing say counts of words seen in a document, is typically sparse (most documents only contain a limited portion of the full vocabulary), a frequentist estimate of word probability will say that certain words can never occur, just because it's never seen them. The Bayesian estimate will still assign some non-zero chance of seeing that word.

Practically speaking, this leads to the idea of "smoothing" in tf-idf (text-frequency-inverse-document-frequency) vectors, by adding 1 to document frequencies. You don't need Bayesian statistics to do this, but maybe you never would have thought of it otherwise!

Re: Introduction to Support Vector Machines in Machine Learning

#29

Post by phunge »

tesseract2 wrote: post #14
astrophysician wrote: post #12

If you're new to ML or datascience, I would recommend working to build a strong basis in Bayesian statistics. It will help you understand how all of the "canonical" ML methods relate to one another, and will give you a basis for building off of them. In particular, aspire to learn probabilistic graphical models + the libraries to train them (like pyro, tensorflow probability, Edward, Stan). They have a steep learning…

Thanks for this insight. Can you kindly also suggest a good book for someone to start with Bayesian Statistics? I could really use a suggestion about first and second book on this. About Probabilistic Graphical Models, is there book other than Daphne Koller's book that you would suggest?

For Bayesian stats, "Statistical Rethinking" by McElreath is a masterpiece.

Re: Introduction to Support Vector Machines in Machine Learning

#30

Post by smbrian »

MaxBarraclough wrote: post #26
smbrian wrote: post #4

Stay away, in my opinion. I spent a year supporting a SVM in a production machine learning application, and it made me wish the ML research community hadn't been so in love with them for so long. They're the perfect blend of theoretically elegant and practically impractical. Training scales as O(n^3), serialized models are heavyweight, prediction is slow. They're like Gaussian Processes, except warped and without any…

So you're saying to stay away from SVMs, rather than to stay away from this particular tutorial?

Sorry, I should've been clearer! Beginner to ML? Stay away from SVMs.

This tutorial looks good, and well written.

Post reply on HN