Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introduction to Support Vector Machines in Machine Learning

ranvir.xyz

Post reply on HN
Print view View original

11–20 of 90 posts

Re: Introduction to Support Vector Machines in Machine Learning

#11

Post by snovv_crash »

Der_Einzige wrote: post #7
smbrian wrote: post #4

Stay away, in my opinion. I spent a year supporting a SVM in a production machine learning application, and it made me wish the ML research community hadn't been so in love with them for so long. They're the perfect blend of theoretically elegant and practically impractical. Training scales as O(n^3), serialized models are heavyweight, prediction is slow. They're like Gaussian Processes, except warped and without any…

Kernel function is simple - Are you in a high dimensional space? If so, choose linear kernel. Else? Choose the most non-linear one you can (usually a guassian or RBF). I suppose quadratic and the other kernals are useful if what your modeling looks like that but in practice that is rare. Prediction is not that slow with linear SVMs especially not compared to something like K-NN. The main hyperparamaters which matter…

> Prediction is not that slow with linear SVMs especially not compared to something like K-NN.

Provided your structural dimensionality is below about 10 (ie. 10 dominant eigenvalues for your features), then KNN can be O(log(N)) for prediction via a well designed Kd-Tree.

KNN is also really simple to understand, and to design features for. It also never really tends to throw up surprises, which for production is the kind of thing you want. Most importantly, the failures tend to 'make sense' to humans, so you stay out of the uncanny valley.

Re: Introduction to Support Vector Machines in Machine Learning

#12

Post by astrophysician »

If you're new to ML or datascience, I would recommend working to build a strong basis in Bayesian statistics. It will help you understand how all of the "canonical" ML methods relate to one another, and will give you a basis for building off of them.

In particular, aspire to learn probabilistic graphical models + the libraries to train them (like pyro, tensorflow probability, Edward, Stan). They have a steep learning curve, especially if you're new to the game, but the reward is great.

All of these methods have their place. SVM's have their place, but also aren't great for probability calibration and non-linear SVM's like every single kernel method can scale absolutely terribly. Neural networks have their place, sometimes as a component of a larger statistical model, sometimes as a feature selector, sometimes in and of themselves. They're also very often the wrong choice for a problem.

Don't fall into the beginner trap: sometimes people tend to mistake 'what is the hottest research topic' for 'what is the right solution to my problem given my constraints, (data limitations, time limitations, skill limitations, etc.)'. Be realistic, don't use magical thinking, and have a strong basis in statistics to weed out the beautiful non-bullshit from the bullshit that is frustratingly prevalent (everyone and their mother is an ML expert today).

EDIT: I want to also clarify: I don't mean to suggest the author is new to ML, I just mean this as general advice for anyone coming here who is new to DS/ML. The article looks great!

Re: Introduction to Support Vector Machines in Machine Learning

#13

Post by rangerranvir »

astrophysician wrote: post #12

If you're new to ML or datascience, I would recommend working to build a strong basis in Bayesian statistics. It will help you understand how all of the "canonical" ML methods relate to one another, and will give you a basis for building off of them. In particular, aspire to learn probabilistic graphical models + the libraries to train them (like pyro, tensorflow probability, Edward, Stan). They have a steep learning…

Thanks for the advice. Will definitely try to follow that. I was trying to learn basics of statistics and went through most of the intro to statistical learning. Will complete the rest in few days.

I am more of a book person, if you have any other resource for probabilistic graphical models, please share here.

Re: Introduction to Support Vector Machines in Machine Learning

#14

Post by tesseract2 »

astrophysician wrote: post #12

If you're new to ML or datascience, I would recommend working to build a strong basis in Bayesian statistics. It will help you understand how all of the "canonical" ML methods relate to one another, and will give you a basis for building off of them. In particular, aspire to learn probabilistic graphical models + the libraries to train them (like pyro, tensorflow probability, Edward, Stan). They have a steep learning…

Thanks for this insight. Can you kindly also suggest a good book for someone to start with Bayesian Statistics? I could really use a suggestion about first and second book on this.

About Probabilistic Graphical Models, is there book other than Daphne Koller's book that you would suggest?

Re: Introduction to Support Vector Machines in Machine Learning

#15

Post by rangerranvir »

stu2b50 wrote: post #10
bitforger wrote: post #2

ITT: Whether SVMs are still relevant in the deep learning era. Some junior researchers will say neural networks are all you need. Industry folks will talk about how they still use decision trees. Personally, I'm quite bullish on the resurgence of SVMs as SOTA. What did it for me was Mikhail Belkin's talk at IAS.[1] [1] https://m.youtube.com/watch?index=15&list=PLdDZb3TwJPZ5dqqg_...

I mean NNs are still quite bad at low n tabular data (and they may always be), which is honestly how a lot of real life data is, so there is clearly a need for not a neural network. I feel like I've seem more tree ensembles in the wild than SVMs, though.

Anyway the idea of NNs was introduced to work on data which a simple human brain couldn't make sense of.

For more general tabular data, like trees, regression and even rule based models are more realistic.

Re: Introduction to Support Vector Machines in Machine Learning

#16

Post by rangerranvir »

tesseract2 wrote: post #14
astrophysician wrote: post #12

If you're new to ML or datascience, I would recommend working to build a strong basis in Bayesian statistics. It will help you understand how all of the "canonical" ML methods relate to one another, and will give you a basis for building off of them. In particular, aspire to learn probabilistic graphical models + the libraries to train them (like pyro, tensorflow probability, Edward, Stan). They have a steep learning…

Thanks for this insight. Can you kindly also suggest a good book for someone to start with Bayesian Statistics? I could really use a suggestion about first and second book on this. About Probabilistic Graphical Models, is there book other than Daphne Koller's book that you would suggest?

+1

Re: Introduction to Support Vector Machines in Machine Learning

#17

Post by uoaei »

astrophysician wrote: post #12

If you're new to ML or datascience, I would recommend working to build a strong basis in Bayesian statistics. It will help you understand how all of the "canonical" ML methods relate to one another, and will give you a basis for building off of them. In particular, aspire to learn probabilistic graphical models + the libraries to train them (like pyro, tensorflow probability, Edward, Stan). They have a steep learning…

Great comments. I heartily agree and support the statement about probabilistic graphical models. Just to add a couple more facets to this perspective:

'State of the art' does not always mean 'best for your task', and in fact lately depending on your field SOTA sometimes simply means 'unaffordable' for anyone whose budget is under 1 million dollars.

Try linear methods first.

Ensembles of decent models are usually good models. The point above about probability calibration can be at least somewhat mitigated by using ensemble averages.

Don't just assume "the $MODEL will figure it out" if you give it shitloads of degrees of freedom. Machine learning efficiency all comes down to efficiency of representation, and feature engineering can achieve huge payoffs if/when you incorporate domain knowledge and expertise.

Once you gain a perspective into the "universality" of statistical methods, optimization, and Bayesian probability theory, your work will become a lot easier to reason about. As an example, try to see if you can explain why least-squares fit results from the assumption that model residuals are normally distributed (and what connections this may have to statistical physics!).

Re: Introduction to Support Vector Machines in Machine Learning

#18

Post by uoaei »

tesseract2 wrote: post #14
astrophysician wrote: post #12

If you're new to ML or datascience, I would recommend working to build a strong basis in Bayesian statistics. It will help you understand how all of the "canonical" ML methods relate to one another, and will give you a basis for building off of them. In particular, aspire to learn probabilistic graphical models + the libraries to train them (like pyro, tensorflow probability, Edward, Stan). They have a steep learning…

Thanks for this insight. Can you kindly also suggest a good book for someone to start with Bayesian Statistics? I could really use a suggestion about first and second book on this. About Probabilistic Graphical Models, is there book other than Daphne Koller's book that you would suggest?

Introduction to Statistical Learning

https://faculty.marshall.usc.edu/gareth-james/ISL/

Elements of Statistical Learning

https://web.stanford.edu/~hastie/ElemStatLearn/

Machine Learning: A Probabilistic Perspective

https://mitpress.mit.edu/books/machine-learning-1

Re: Introduction to Support Vector Machines in Machine Learning

#19

Post by astrophysician »

rangerranvir wrote: post #13
astrophysician wrote: post #12

If you're new to ML or datascience, I would recommend working to build a strong basis in Bayesian statistics. It will help you understand how all of the "canonical" ML methods relate to one another, and will give you a basis for building off of them. In particular, aspire to learn probabilistic graphical models + the libraries to train them (like pyro, tensorflow probability, Edward, Stan). They have a steep learning…

Thanks for the advice. Will definitely try to follow that. I was trying to learn basics of statistics and went through most of the intro to statistical learning. Will complete the rest in few days. I am more of a book person, if you have any other resource for probabilistic graphical models, please share here.

Nice! There are many books that cover this, even the docs for Pyro/other libraries are useful, it just depends on your preference for how material is presented + your background.

Bishop's "Pattern Recognition and Machine Learning" has a chapter on PGM's that's free online: https://www.microsoft.com/en-us/research/wp-content/uploads/...

Murphy's "Machine Learning: A Probabilistic Perspective" is another behemoth that covers this stuff, but it's really just your preference.

I say "aspire" because (1) depending on your background, it will likely be something that takes awhile to internalize and really understand, and you will probably realize many times over that you thought you understood something that you actually didn't (2) by learning PGM's, you learn a lot of Bayesian statistics as a side effect, hence why even learning a little bit about them is rewarding.

Once you learn a bit, I would use Pyro/other libraries and try to actually build PGM's for toy problems (or non-toy problems too..) because (1) it will force you to admit to yourself that you don't understand something, (2) the documentation for a lot of these libraries is also useful learning material, and (3) you will see once you learn these libraries that it is fairly easy to do something that would be astoundingly complex if you were to try and do it by hand.

You can basically build most standard ML algorithms as a PGM, so e.g. you can try to do logistic regression as a PGM and compare the results to scikit-learn.

Re: Introduction to Support Vector Machines in Machine Learning

#20

Post by astrophysician »

tesseract2 wrote: post #14
astrophysician wrote: post #12

If you're new to ML or datascience, I would recommend working to build a strong basis in Bayesian statistics. It will help you understand how all of the "canonical" ML methods relate to one another, and will give you a basis for building off of them. In particular, aspire to learn probabilistic graphical models + the libraries to train them (like pyro, tensorflow probability, Edward, Stan). They have a steep learning…

Thanks for this insight. Can you kindly also suggest a good book for someone to start with Bayesian Statistics? I could really use a suggestion about first and second book on this. About Probabilistic Graphical Models, is there book other than Daphne Koller's book that you would suggest?

I think PGM's are covered by a lot of "standard" ML texts -- someone else mentioned Murphy's book which is great and is humongous but is a good reference for pretty much every method under the sun.

Bishop's Pattern Recognition and Machine Learning has a chapter thats free online: https://www.microsoft.com/en-us/research/wp-content/uploads/...

Post reply on HN