A Quick Look at Support Vector Machines • Hacker News Board

Re: A Quick Look at Support Vector Machines

#11

Post by emcq » Thu, Mar 23, 2017, 5:48 PM UTC

syntaxing wrote: post #3

One thing that I discovered recently which surprised me (while taking the Udacity SDC)is how effective and resilient these "older" ML algorithms can be. Neural networks was always my go to method for most of my classification or regression problems for my small side projects. But now I learned with the minimal dataset I have (<5K samples), linear regression, SVM, or decision tress is the way to go. I got higher accur…

If you have good features there is little advantage to a complex model.

In production ML there are still many applications for random forests, linear models or svms. Though I prefer random forests because they require less preprocessing, are super fast to train, and can be easy to explain feature importances.

Re: A Quick Look at Support Vector Machines

#12

Post by jwr » Thu, Mar 23, 2017, 6:48 PM UTC

nafizh wrote: post #9

Aaah, I was hoping for an explanation of the kernel trick. I think that is the hardest concept in support vector machines.

I think I can help with that.

The article nicely explains the data transformation so that it becomes linearly separable. But the trick to the kernel trick is no to transform the data at all.

What you do is use a learning algorithm that doesn't need individual input vectors, but instead only needs their dot products. You then imagine a magical high-dimensional space where your data is (you suppose) linearly separable. The trick is that you never actually transform your data to that magical space — you don't need the input vectors, remember? You only need their dot products. So you define a function that given two vectors in your normal input space returns a scalar. Assuming your function behaves in a sane way (go read about the required properties if you need to), you can think of this function as a dot product. In some kind of magical space — you don't actually care much. You will never transform your data, it might not even be possible to: the most common gaussian kernel is defined over an infinite-dimensional space. But hey, who cares? You take your SVM, give it your kernel function and input data, and off it goes, working as usual, except your dot products are no longer computed in your input space, but in your magical infinite-dimensional space.

It's both really clever and really simple.

Re: A Quick Look at Support Vector Machines

#13

Post by Scea91 » Thu, Mar 23, 2017, 6:57 PM UTC

emcq wrote: post #11

syntaxing wrote: post #3

One thing that I discovered recently which surprised me (while taking the Udacity SDC)is how effective and resilient these "older" ML algorithms can be. Neural networks was always my go to method for most of my classification or regression problems for my small side projects. But now I learned with the minimal dataset I have (<5K samples), linear regression, SVM, or decision tress is the way to go. I got higher accur…

If you have good features there is little advantage to a complex model. In production ML there are still many applications for random forests, linear models or svms. Though I prefer random forests because they require less preprocessing, are super fast to train, and can be easy to explain feature importances.

In addition, random forests often work very well out-of-the-box with 'default' hyperparameter settings.

Re: A Quick Look at Support Vector Machines

#14

Post by LeanderK » Thu, Mar 23, 2017, 7:32 PM UTC

well, that really was a quick look. Any reading-recommendations about the kernel-functions? How do they work and why are they fast?

Re: A Quick Look at Support Vector Machines

#15

Post by rs86 » Thu, Mar 23, 2017, 7:33 PM UTC

Amazingly well written. Short and to the point, humbly sharing something cool!

Re: A Quick Look at Support Vector Machines

#16

Post by rs86 » Thu, Mar 23, 2017, 7:35 PM UTC

curiousgal wrote: post #10

Many aspects of Machine Learning boil down to optimization problems.

Well all of them do. In ML we always try to select the best description for a dataset, and that involves minimizing some function that represents some kind of goodness of fit

Re: A Quick Look at Support Vector Machines

#17

Post by platz » Thu, Mar 23, 2017, 7:41 PM UTC

akyu wrote: post #8

syntaxing wrote: post #3

One thing that I discovered recently which surprised me (while taking the Udacity SDC)is how effective and resilient these "older" ML algorithms can be. Neural networks was always my go to method for most of my classification or regression problems for my small side projects. But now I learned with the minimal dataset I have (<5K samples), linear regression, SVM, or decision tress is the way to go. I got higher accur…

Yes SVMs are still great models. The advantage neural nets have over them are that they can do automatic feature extraction. By the time you get to the last layer of a neural net, you are basically just doing a simple logistic classification, but the features coming in have been learned from all of the previous layers. I've even seen people use pretrained ImageNet classifiers, chop off the last layer and use an SVM a…

> automatic feature extraction

Hope you have a *ton of data, otherwise it's not gonna happen

Re: A Quick Look at Support Vector Machines

#18

Post by radarsat1 » Thu, Mar 23, 2017, 7:51 PM UTC

syntaxing wrote: post #3

One thing that I discovered recently which surprised me (while taking the Udacity SDC)is how effective and resilient these "older" ML algorithms can be. Neural networks was always my go to method for most of my classification or regression problems for my small side projects. But now I learned with the minimal dataset I have (<5K samples), linear regression, SVM, or decision tress is the way to go. I got higher accur…

I'm curious where the idea that SVM are "older" than neural networks comes from. The SVM wikipedia page claims that they were published by Vapnik & Chervonenkis in 1963, while Neural Networks date back at least to Rosenblatt's work in 1958, if not before.

Re: A Quick Look at Support Vector Machines

#19

Post by trendia » Thu, Mar 23, 2017, 7:53 PM UTC

platz wrote: post #17

akyu wrote: post #8

Earlier quoted context omitted.

Yes SVMs are still great models. The advantage neural nets have over them are that they can do automatic feature extraction. By the time you get to the last layer of a neural net, you are basically just doing a simple logistic classification, but the features coming in have been learned from all of the previous layers. I've even seen people use pretrained ImageNet classifiers, chop off the last layer and use an SVM a…

> automatic feature extraction Hope you have a *ton of data, otherwise it's not gonna happen

And a lot of tweaking of configuration parameters until it's "automatic".

Re: A Quick Look at Support Vector Machines

#20

Post by shas3 » Thu, Mar 23, 2017, 8:17 PM UTC

Very cool! However, I think the author should have spent a a few more words and figures to distinguish support vector machines from standard perceptrons. Maximum margin classification and the definition of 'support vectors,' in my experience, helps demystify the algorithm.