Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introduction to Support Vector Machines in Machine Learning

ranvir.xyz

Post reply on HN
Print view View original

61–70 of 90 posts

Re: Introduction to Support Vector Machines in Machine Learning

#61

Post by rangerranvir »

kmundnic wrote: post #24
uoaei wrote: post #18

Earlier quoted context omitted.

Introduction to Statistical Learning https://faculty.marshall.usc.edu/gareth-james/ISL/ Elements of Statistical Learning https://web.stanford.edu/~hastie/ElemStatLearn/ Machine Learning: A Probabilistic Perspective https://mitpress.mit.edu/books/machine-learning-1

"Machine Learning: a Probabilistic Perspective" is more an encyclopedia of algorithms I would say, and it has lots of typos. I personally would not recommend it (except for the amount of algorithms that it covers, many of which are usually not found in other books).

Thanks for early warning. Will have to keep that in mind.

Re: Introduction to Support Vector Machines in Machine Learning

#62

Post by rangerranvir »

smbrian wrote: post #30
MaxBarraclough wrote: post #26

Earlier quoted context omitted.

So you're saying to stay away from SVMs, rather than to stay away from this particular tutorial?

Sorry, I should've been clearer! Beginner to ML? Stay away from SVMs. This tutorial looks good, and well written.

Thanks for saying that. Means a lot.

Re: Introduction to Support Vector Machines in Machine Learning

#63

Post by rangerranvir »

starchild_3001 wrote: post #44

I've been an ML practioner since 2009. I've used every method imaginable or popular, I think. With the exception of non-linear SVMs. Linear SVM => All good, just the hingle loss optimization. Non-linear SVM, a bit of overkill with basis expansion. Just too slow, or too complex a model? My impression: SVMs are more of theoretical interest than practical interest. Yeah, learn your statistics. Loss functions. Additive m…

We can definitely learn a piece from such an experienced practitioner. Thanks for sharing, I think your intuition matches with the other experienced once in the comments.

Re: Introduction to Support Vector Machines in Machine Learning

#64

Post by KKKKkkkk1 »

astrophysician wrote: post #53
unishark wrote: post #37

Earlier quoted context omitted.

Personally I'd advise against both SVM's and Bayesian methods for a beginner. Bayesian statistics is very much the deep end of the pool. Graphical models and Bayesian methods generally may make a comeback but such approaches have been superseded by other methods for good reasons, i.e. scaling. A strong basis in statistics is certainly a great thing, but that can be maximum likelihood plus Bayes law (i.e. "MAP" estima…

> Personally I'd advise against both SVM's and Bayesian methods for a beginner. Bayesian statistics is very much the deep end of the pool. I don’t know, I think it depends on what you mean by Bayesian. I would say understanding loss functions and regularization requires some understanding of Bayesian stats (just knowing that it comes from log p(x|q) + log p(q) and what both of those terms mean). > Graphical models an…

How does understanding loss functions and regularization require understanding Bayesian statistics? Those notions are literally part of linear regression theory.

Re: Introduction to Support Vector Machines in Machine Learning

#65

Post by ssivark »

KKKKkkkk1 wrote: post #64
astrophysician wrote: post #53

Earlier quoted context omitted.

> Personally I'd advise against both SVM's and Bayesian methods for a beginner. Bayesian statistics is very much the deep end of the pool. I don’t know, I think it depends on what you mean by Bayesian. I would say understanding loss functions and regularization requires some understanding of Bayesian stats (just knowing that it comes from log p(x|q) + log p(q) and what both of those terms mean). > Graphical models an…

How does understanding loss functions and regularization require understanding Bayesian statistics? Those notions are literally part of linear regression theory.

Only if you use them as a black box. What is considered l2 norm in regression theory corresponds to a Gaussian likelihood, whose log-likelihood becomes quadratic. And one typically might not have any priors (which is subtly misleading, because the "flat" prior is highly dependent on the chosen representation for the underlying degree of freedom). Note that regularizations are just log-priors.

Without an understanding of the underlying Bayesian formulation, linear regression theory might look like a vast and somewhat ad-hoc collection of separate ideas (eg: robust loss functions, and the many different kinds of regularizations), but seen in the correct light, it is easy to start with a general formulation and specialize it nicely to your problem. Working that way, you can often design a good solution for your problem without searching through handbooks for possible pre-defined methods. You can also combine multiple ideas very easily. eg: A couple of weeks ago, working from first principles I rediscovered what is called "Lavrentyev_regularization": https://en.wikipedia.org/wiki/Tikhonov_regularization#Lavren...

Re: Introduction to Support Vector Machines in Machine Learning

#66

Post by ssivark »

astrophysician wrote: post #53
unishark wrote: post #37

Earlier quoted context omitted.

Personally I'd advise against both SVM's and Bayesian methods for a beginner. Bayesian statistics is very much the deep end of the pool. Graphical models and Bayesian methods generally may make a comeback but such approaches have been superseded by other methods for good reasons, i.e. scaling. A strong basis in statistics is certainly a great thing, but that can be maximum likelihood plus Bayes law (i.e. "MAP" estima…

> Personally I'd advise against both SVM's and Bayesian methods for a beginner. Bayesian statistics is very much the deep end of the pool. I don’t know, I think it depends on what you mean by Bayesian. I would say understanding loss functions and regularization requires some understanding of Bayesian stats (just knowing that it comes from log p(x|q) + log p(q) and what both of those terms mean). > Graphical models an…

To elaborate on the point: When doing probabilistic modeling, whether one realizes or not, there typically is an underlying Bayesian formulation which explains what one is doing. Now, that might be well-aligned with the problem of interest (or not), and being clear on the fundamentals helps understand that, and also to compose distinct ideas which make sense together in the context of the problem. eg: see my comments below, in the context of linear regression from a Bayesian perspective.

Also, while "scaling" with data is a very hip thing these days, for most problems of interest it is very difficult/expensive to get lots of data (or afford compute). Further, humans often have very useful domain-models which are worth encoding into the structure of the model. This also helps nicely mix together a conventional "software" modeling with probabilistic aspects (for those who weren't aware, this flows towards what is called "probabilistic programming", and recent developments have made significant progress towards methods which work for an "intermediate" dimensionality, if not "large" dimensionality).

--

@astrophysician: Feels nice to see expressed so clearly, a perspective that I share! Feel free to get in touch if you'd like to discuss ML.

Re: Introduction to Support Vector Machines in Machine Learning

#67

Post by sriram_malhar »

astrophysician wrote: post #12

If you're new to ML or datascience, I would recommend working to build a strong basis in Bayesian statistics. It will help you understand how all of the "canonical" ML methods relate to one another, and will give you a basis for building off of them. In particular, aspire to learn probabilistic graphical models + the libraries to train them (like pyro, tensorflow probability, Edward, Stan). They have a steep learning…

Can you please suggest some resources for a math-comfortable beginner?

Re: Introduction to Support Vector Machines in Machine Learning

#68

Post by prox »

ssivark wrote: post #66
astrophysician wrote: post #53

Earlier quoted context omitted.

> Personally I'd advise against both SVM's and Bayesian methods for a beginner. Bayesian statistics is very much the deep end of the pool. I don’t know, I think it depends on what you mean by Bayesian. I would say understanding loss functions and regularization requires some understanding of Bayesian stats (just knowing that it comes from log p(x|q) + log p(q) and what both of those terms mean). > Graphical models an…

To elaborate on the point: When doing probabilistic modeling, whether one realizes or not, there typically is an underlying Bayesian formulation which explains what one is doing. Now, that might be well-aligned with the problem of interest (or not), and being clear on the fundamentals helps understand that, and also to compose distinct ideas which make sense together in the context of the problem. eg: see my comments…

Is there a good book for a beginner to get acquainted with ML? I have experience in Python, JS if that helps narrow it down.
Post reply on HN