Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introduction to Support Vector Machines in Machine Learning

ranvir.xyz

Post reply on HN
Print view View original

51–60 of 90 posts

Re: Introduction to Support Vector Machines in Machine Learning

#51

Post by astrophysician »

astrofinch wrote: post #48
unishark wrote: post #37

Earlier quoted context omitted.

Personally I'd advise against both SVM's and Bayesian methods for a beginner. Bayesian statistics is very much the deep end of the pool. Graphical models and Bayesian methods generally may make a comeback but such approaches have been superseded by other methods for good reasons, i.e. scaling. A strong basis in statistics is certainly a great thing, but that can be maximum likelihood plus Bayes law (i.e. "MAP" estima…

>The training algorithms for deep learning are also the hottest algorithm research area in machine learning, and are certainly applicable beyond deep learning. The lore I've heard is that most new deep learning training algorithms (optimization algorithms) only work better on particular special cases, and it is hard to do better than the established algorithms in general. I'm also not sure why you're saying they're a…

Adam, RMSProp, etc are just flavors of gradient descent so they’re useful on anything from ResNet to logistic regression. There are more flavors like natural gradient that are more useful for smaller problems since they require a Hessian matrix, but gradient descent is gradient descent. We use Adam in production for logistic regression, not for any particular reason really, just happens to work.

Re: Introduction to Support Vector Machines in Machine Learning

#52

Post by ma2rten »

bitforger wrote: post #2

ITT: Whether SVMs are still relevant in the deep learning era. Some junior researchers will say neural networks are all you need. Industry folks will talk about how they still use decision trees. Personally, I'm quite bullish on the resurgence of SVMs as SOTA. What did it for me was Mikhail Belkin's talk at IAS.[1] [1] https://m.youtube.com/watch?index=15&list=PLdDZb3TwJPZ5dqqg_...

I don't have time watch the video could you summarize why you think SVMs will become SOTA and on which problem?

Re: Introduction to Support Vector Machines in Machine Learning

#53

Post by astrophysician »

unishark wrote: post #37
astrophysician wrote: post #12

If you're new to ML or datascience, I would recommend working to build a strong basis in Bayesian statistics. It will help you understand how all of the "canonical" ML methods relate to one another, and will give you a basis for building off of them. In particular, aspire to learn probabilistic graphical models + the libraries to train them (like pyro, tensorflow probability, Edward, Stan). They have a steep learning…

Personally I'd advise against both SVM's and Bayesian methods for a beginner. Bayesian statistics is very much the deep end of the pool. Graphical models and Bayesian methods generally may make a comeback but such approaches have been superseded by other methods for good reasons, i.e. scaling. A strong basis in statistics is certainly a great thing, but that can be maximum likelihood plus Bayes law (i.e. "MAP" estima…

> Personally I'd advise against both SVM's and Bayesian methods for a beginner. Bayesian statistics is very much the deep end of the pool.

I don’t know, I think it depends on what you mean by Bayesian. I would say understanding loss functions and regularization requires some understanding of Bayesian stats (just knowing that it comes from log p(x|q) + log p(q) and what both of those terms mean).

> Graphical models and Bayesian methods generally may make a comeback but such approaches have been superseded by other methods for good reasons, i.e. scaling

Can you be more specific here? It sounds like you’re talking about a particular problem or class of methods. PGMs/Bayesian methods can mean basically anything from logistic regression to running HMC on some hierarchical model using 10,000 CPU hours. I just mean aspiring to learn PGMs will force you to quickly learn and gain a deeper understanding of and appreciation for Bayesian stats, and then you can build on that for years and years. But it depends on what you’re interested in doing —- there’s a difference between model building and inference; you can spend your whole life using the same loss function and just focus on making your NN architecture better, you don’t need much Bayesian stats to do that.

> i.e. "MAP" estimation which is more of a hack to ML than an actual Bayesian method

Huh? Maybe we mean different things by Bayesian — the mode of your posterior seems pretty Bayesian to me!

> Meanwhile a strong basis in "deterministic methods", as an alternative way to spend that learning effort, has its own rewards. The training algorithms for deep learning are also the hottest algorithm research area in machine learning, and are certainly applicable beyond deep learning. For that matter a thorough understanding of SVM delves into convex optimization, an extremely powerful framework as well.

Would agree that optimization is an important part of ML/DS, but since nowadays virtually all of the most popular optimization algorithms are available at our fingertips in e.g. pytorch, I would still think its better to start with trying to build a fundamental understanding of how to frame problems. But that’s colored by my own experience and background, people’s priorities should be different depending on what they want to do.

Re: Introduction to Support Vector Machines in Machine Learning

#54

Post by nightski »

phunge wrote: post #29
tesseract2 wrote: post #14

Earlier quoted context omitted.

Thanks for this insight. Can you kindly also suggest a good book for someone to start with Bayesian Statistics? I could really use a suggestion about first and second book on this. About Probabilistic Graphical Models, is there book other than Daphne Koller's book that you would suggest?

For Bayesian stats, "Statistical Rethinking" by McElreath is a masterpiece.

This should be the top comment, I'm reading his newly released 2nd edition and it is outstanding. Other mentions are -

  - Doing Bayesian Data Analysis (dog book)
  - Student's Guide to Bayesian Statistics
Slightly more advanced

  - Bayesian Data Analysis 3 (currently free! http://www.stat.columbia.edu/~gelman/book/)

Re: Introduction to Support Vector Machines in Machine Learning

#55

Post by unishark »

astrofinch wrote: post #48
unishark wrote: post #37

Earlier quoted context omitted.

Personally I'd advise against both SVM's and Bayesian methods for a beginner. Bayesian statistics is very much the deep end of the pool. Graphical models and Bayesian methods generally may make a comeback but such approaches have been superseded by other methods for good reasons, i.e. scaling. A strong basis in statistics is certainly a great thing, but that can be maximum likelihood plus Bayes law (i.e. "MAP" estima…

>The training algorithms for deep learning are also the hottest algorithm research area in machine learning, and are certainly applicable beyond deep learning. The lore I've heard is that most new deep learning training algorithms (optimization algorithms) only work better on particular special cases, and it is hard to do better than the established algorithms in general. I'm also not sure why you're saying they're a…

I'd more generally describe the area as first order optimization, including methods like acceleration, automatic differentiation, stochastic approaches. Adam is just one trick for determining a hyperparameter.

They are usable everywhere derivative-based optimization is usable. Which certainly means SVM's, though since it's a shallow method you don't need much data to train it, and hence don't need a scalable optimization methods (it would just be unnecessarily slow). But you certainly could do it if you somehow needed to. Here's the first hit on google for "sgd svm': https://scikit-learn.org/stable/modules/generated/sklearn.li...

The fact that you can't use first order optimization methods for graphical models is one answer to the question of why everyone doesn't use them. Though for small models there are deep networks which model them and are trained as per usual for neural networks. I think this is still an active research area.

Re: Introduction to Support Vector Machines in Machine Learning

#56

Post by solar_perplexus »

tesseract2 wrote: post #14
astrophysician wrote: post #12

If you're new to ML or datascience, I would recommend working to build a strong basis in Bayesian statistics. It will help you understand how all of the "canonical" ML methods relate to one another, and will give you a basis for building off of them. In particular, aspire to learn probabilistic graphical models + the libraries to train them (like pyro, tensorflow probability, Edward, Stan). They have a steep learning…

Thanks for this insight. Can you kindly also suggest a good book for someone to start with Bayesian Statistics? I could really use a suggestion about first and second book on this. About Probabilistic Graphical Models, is there book other than Daphne Koller's book that you would suggest?

I took a course on Applied Bayesian Statistics taught by David Draper in grad school and we covered Bayesian Data Analysis (Gelman et Al.) http://www.stat.columbia.edu/~gelman/book/ and Probability Theory and tbe Logic of Science by Ed Jaynes: https://www.amazon.com/dp/0521592712/ref=cm_sw_r_em_apa_i_v3...

The former is a much recommended book since it's very comprehensive and builds everything from the ground up and was the basis for the entire course. The latter is a beast of it's own and we simply covered what was effectively the first chapter as part of the course.

Re: Introduction to Support Vector Machines in Machine Learning

#57

Post by rangerranvir »

unishark wrote: post #55
astrofinch wrote: post #48

Earlier quoted context omitted.

>The training algorithms for deep learning are also the hottest algorithm research area in machine learning, and are certainly applicable beyond deep learning. The lore I've heard is that most new deep learning training algorithms (optimization algorithms) only work better on particular special cases, and it is hard to do better than the established algorithms in general. I'm also not sure why you're saying they're a…

I'd more generally describe the area as first order optimization, including methods like acceleration, automatic differentiation, stochastic approaches. Adam is just one trick for determining a hyperparameter. They are usable everywhere derivative-based optimization is usable. Which certainly means SVM's, though since it's a shallow method you don't need much data to train it, and hence don't need a scalable optimiza…

Thanks for sharing, looks promising.

Re: Introduction to Support Vector Machines in Machine Learning

#58

Post by unishark »

astrophysician wrote: post #53
unishark wrote: post #37

Earlier quoted context omitted.

Personally I'd advise against both SVM's and Bayesian methods for a beginner. Bayesian statistics is very much the deep end of the pool. Graphical models and Bayesian methods generally may make a comeback but such approaches have been superseded by other methods for good reasons, i.e. scaling. A strong basis in statistics is certainly a great thing, but that can be maximum likelihood plus Bayes law (i.e. "MAP" estima…

> Personally I'd advise against both SVM's and Bayesian methods for a beginner. Bayesian statistics is very much the deep end of the pool. I don’t know, I think it depends on what you mean by Bayesian. I would say understanding loss functions and regularization requires some understanding of Bayesian stats (just knowing that it comes from log p(x|q) + log p(q) and what both of those terms mean). > Graphical models an…

By Bayesian methods I mean methods which also describe everything with statistical distributions including hyperparameters, and which solve for a distribution (or some hard-to-get feature of it like its mean or confidence intervals), instead of just a maximum probability estimate where you can make shortcuts.

For example Ridge regression where you put a gaussian prior on the predictor then guess at its variance as a hyperparameter for a quadratic penalty (with cross validation or something) is MAP but not actually "Bayesian". Bayesians also put a prior distribution on the hyperparameter itself making life extra difficult.

I agree it's an awkward definition but it does differentiate between the easy tricks and the tough approaches Bayesian researchers actually work on.

Even in deep learning it's simple to put a penalty term on the loss or weights for that matter, so you could call it a MAP estimate. That's implemented everywhere. For that matter you can claim any neural net classifier with a sigmoid or softmax at the final layer is doing logistic regression. With multilayer networks just adding a representation learning stage to it.

As I said cheers to statistics, just that stopping at MAP estimates will cover most everything. If one goes looking for a Bayesian methods text or review article it will be well beyond that.

By scaling problems I basically the fact that monte carlo doesn't work in large numbers of dimensions, where the parameters of the joint distribution of everything are vast.

Re: Introduction to Support Vector Machines in Machine Learning

#59

Post by rangerranvir »

astrophysician wrote: post #51
astrofinch wrote: post #48

Earlier quoted context omitted.

>The training algorithms for deep learning are also the hottest algorithm research area in machine learning, and are certainly applicable beyond deep learning. The lore I've heard is that most new deep learning training algorithms (optimization algorithms) only work better on particular special cases, and it is hard to do better than the established algorithms in general. I'm also not sure why you're saying they're a…

Adam, RMSProp, etc are just flavors of gradient descent so they’re useful on anything from ResNet to logistic regression. There are more flavors like natural gradient that are more useful for smaller problems since they require a Hessian matrix, but gradient descent is gradient descent. We use Adam in production for logistic regression, not for any particular reason really, just happens to work.

Random question, do you guys have tried linear regression in place of Adam and was there a significant change in the training time?

And on what data you decide to move on to NN based model rather than using a simpler models like linear regression etc?

Re: Introduction to Support Vector Machines in Machine Learning

#60

Post by rangerranvir »

rjvir wrote: post #27
astrophysician wrote: post #12

If you're new to ML or datascience, I would recommend working to build a strong basis in Bayesian statistics. It will help you understand how all of the "canonical" ML methods relate to one another, and will give you a basis for building off of them. In particular, aspire to learn probabilistic graphical models + the libraries to train them (like pyro, tensorflow probability, Edward, Stan). They have a steep learning…

Different people learn in different ways, but personally I’ve had more success with the opposite approach, ie “top-down”. As in, rather than learning in depth all the low level parts then finally putting it together at the end, start with a surface high-level understanding of a working prototype then expand into the details of how everything works inside. In the case of ML, this could mean starting with a 5 line SciK…

Great Point! I do read article where someone builds such cool stuff every now and then to keep in mind what's coming.

Anyway whatever works. Ultimate aim is to learn and have fun.

Post reply on HN