Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Introduction to Support Vector Machines

monkeylearn.com

Post reply on HN
Print view View original

51–60 of 64 posts

Re: An Introduction to Support Vector Machines

#51

Post by IanCal »

sivvy wrote: post #38
IanCal wrote: post #35

Earlier quoted context omitted.

The simplest is to pop it on the top. Run you DNN to reduce your input down to a nicer cleaner smaller dimensional output, then plop an SVM on top for classification.

Seems like in that case you would train both models separately on different cost functions. By phrasing it as a layer I was expecting both the SVM and the DNN could be trained simultaneously.

Unless things have changed, one of the key benefits of DNNs was that you trained them layer by layer.

You also want to be able to train the DNN on your unlabelled data and the SVM on your much smaller labelled set.

Re: An Introduction to Support Vector Machines

#52

Post by foota »

currymj wrote: post #22

Since neural nets are winning at the moment, it's easy to see SVMs as an underdog, being ignored due to deep learning hype and PR. This is kind of true, but it's worth noting that 10-15 years ago we had the exact opposite situation. Neural nets were a once promising technique that had stagnated/hit their limits, while SVMs were the new state of the art. People were coming up with dozens of unnecessary variations on t…

Would you say machine learning teens to overfit?

Re: An Introduction to Support Vector Machines

#53

Post by Longwelwind »

tgeery wrote: post #47
Longwelwind wrote: post #32

Earlier quoted context omitted.

In the decision function of an SVM, you compute the scalar products of the support vectors (points that are on the margin of your hyperplane, or more precisely, the points that constrain your hyperplane) and your new sample point: x· sv The "z" the article defines is a new component that will be taken into account in the scalar product. A more mathematical way of seeing that is that you define a function phi that tak…

Thank you. This was an amazing explanation. I am new to SVM's but did not make the connection that margin points (observations along the margin of the hyperplane) become your support vectors. This makes a lot more sense. And if I am following correctly, it would make sense that the final step would then be: We would maximize the dot product of a new observation with the support vectors to determine its classification…

During the learning phase of the SVM, you try to find an hyperplane that maximizes the margin.

The decision function of an SVM can be written as:

  f(x) = sign(sum alpha_sv y_sv k(x, sv))
Where sum represents the sum over all support vectors "sv", "y_sv" represents the class of the sample (red=1, blue=-1, for example), "alpha_sv" is the result of the optimization in the learning phase during the learning phase (it is equal to zero for a point that is not a support vector, and is positive otherwise).

The decision function is a sum over all support vectors balanced by the "k" function (that can thus be seen a similarity function between 2 points in your kernel), the y_i will make the term positive or negative depending on the class of the support vector. You take the sign of this sum (1 -> red, -1 -> blue, in our example), and it gives you the predicted class of your sample.

Re: An Introduction to Support Vector Machines

#55

Post by JustFinishedBSG »

deepnotderp wrote: post #6

I'd just like to note that instead of creating additional animosity between SVMs and deep nets, you could use both together. SVMs with hinge loss can be Yet-another-layer (tm) in your deep net, to be used when it provides better performance.

Yes

https://arxiv.org/abs/1605.06265

http://papers.nips.cc/paper/5348-convolutional-kernel-networ...

Re: An Introduction to Support Vector Machines

#56

Post by JustFinishedBSG »

strebler wrote: post #13
deepnotderp wrote: post #6

I'd just like to note that instead of creating additional animosity between SVMs and deep nets, you could use both together. SVMs with hinge loss can be Yet-another-layer (tm) in your deep net, to be used when it provides better performance.

That's a great point. Fundamentally, if you look at something like a CNN, what it's really doing is producing a feature descriptor based on the input image. One can easily use that feature descriptor in a classic SVM, alongside (or instead of) SoftMax.

You may be interested by

https://arxiv.org/abs/1605.06265

http://papers.nips.cc/paper/5348-convolutional-kernel-networ...

Re: An Introduction to Support Vector Machines

#57

Post by max_ »

currymj wrote: post #22

Since neural nets are winning at the moment, it's easy to see SVMs as an underdog, being ignored due to deep learning hype and PR. This is kind of true, but it's worth noting that 10-15 years ago we had the exact opposite situation. Neural nets were a once promising technique that had stagnated/hit their limits, while SVMs were the new state of the art. People were coming up with dozens of unnecessary variations on t…

>People were coming up with dozens of unnecessary variations on them, everybody in the world was trying to shoehorn the word "kernel" into their paper titles, using some kind of kernel method was a surefire way to get published.

I took Andrej Karpthy's tutorial on Nueral net and learned that an SVM is basically one neuron.

http://karpathy.github.io/neuralnets/

Re: An Introduction to Support Vector Machines

#58

Post by foota »

foota wrote: post #52
currymj wrote: post #22

Since neural nets are winning at the moment, it's easy to see SVMs as an underdog, being ignored due to deep learning hype and PR. This is kind of true, but it's worth noting that 10-15 years ago we had the exact opposite situation. Neural nets were a once promising technique that had stagnated/hit their limits, while SVMs were the new state of the art. People were coming up with dozens of unnecessary variations on t…

Would you say machine learning teens to overfit?

Ugh too late to edit, but that was supposed to be tends.

Re: An Introduction to Support Vector Machines

#59

Post by CuriouslyC »

thearn4 wrote: post #31
kefka wrote: post #29

Earlier quoted context omitted.

> I wish machine learning research didn't respond so strongly to trends and hype, It's really because nobody actually understands what's going on inside a ML algorithm. When you give it a ginormous dataset, what data is it really using to make its determination of [0.0000999192346 , .91128756789 , 0 , .62819364 , 32.8172] Because what I do for ML is do a supervised fit, then use a next to test and confirm fitness, th…

Shouldn't it be possible to backpropogate those categorical outputs all the way back to the inputs/features (NOT weights) after a forward pass, to localize the sensitivity of them with respect to the actual pixels for a prediction? I imagine that would have to give at least some insight. Beyond that, the convolution/max pool repeated steps could be understood to be applying something akin to a multi-level wavelet dec…

Bagging and bootstrap ensemble methods aren't really that confusing. Just think of it as stochastic gradient descent on a much larger hypothetical data set.

The effect is same one that occurs when you get a group of people together to estimate the number of jelly beans in a jar. All the estimators are biased, but if that bias is drawn from a zero mean distribution, deviation of the average bias goes down as the number of estimators increases.

Re: An Introduction to Support Vector Machines

#60

Post by CuriouslyC »

Asdfbla wrote: post #14

I remember that only a few years ago, in a computational statistics class I took the lecturer mentioned how SVMs (and Random Forests) have largely replaced neural networks. How things can change so quickly... I always liked SVMs for the elegance of the kernel trick, but I guess choosing the right kernel functions and parameters for them wasn't that much easier than training a neural net either.

If you like SVMs, you should check out gaussian processes (GP). They work with covariance kernels similar to SVM, but the result is fully Bayesian. With most modern GP packages you can even set priors on your kernel and mean functions, then use either optimization or markov chain monte carlo to select optimal values.

The only downside to GPs is that they are O(N^3) in time, so not applicable to big data. There are stochastic GPs that approximate using batch learning, but they're not as polished.

Post reply on HN