Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Introduction to Support Vector Machines

monkeylearn.com

Post reply on HN
Print view View original

31–40 of 64 posts

Re: An Introduction to Support Vector Machines

#31

Post by thearn4 »

kefka wrote: post #29
currymj wrote: post #22

Since neural nets are winning at the moment, it's easy to see SVMs as an underdog, being ignored due to deep learning hype and PR. This is kind of true, but it's worth noting that 10-15 years ago we had the exact opposite situation. Neural nets were a once promising technique that had stagnated/hit their limits, while SVMs were the new state of the art. People were coming up with dozens of unnecessary variations on t…

> I wish machine learning research didn't respond so strongly to trends and hype, It's really because nobody actually understands what's going on inside a ML algorithm. When you give it a ginormous dataset, what data is it really using to make its determination of [0.0000999192346 , .91128756789 , 0 , .62819364 , 32.8172] Because what I do for ML is do a supervised fit, then use a next to test and confirm fitness, th…

Shouldn't it be possible to backpropogate those categorical outputs all the way back to the inputs/features (NOT weights) after a forward pass, to localize the sensitivity of them with respect to the actual pixels for a prediction? I imagine that would have to give at least some insight.

Beyond that, the convolution/max pool repeated steps could be understood to be applying something akin to a multi-level wavelet decomposition, which is pretty well understood. It's how classical matched filtering, Haar cascading, and a wide variety of proceeding image classification methods operated at their first steps too.

CNNs/Deep learning really doesn't seem like a black box at all when examined in sequence. But to me at least, randomized ensemble methods (random forest, etc.) are actually a bit more mysterious to me in their performance out of the box, with little tuning.

Re: An Introduction to Support Vector Machines

#32

Post by Longwelwind »

tgeery wrote: post #20
chrischen wrote: post #9

Can someone explain this part: Imagine the new space we want: z = x² + y² Figure out what the dot product in that space looks like: a · b = xa · xb + ya · yb + za · zb a · b = xa · xb + ya · yb + (xa² + ya²) · (xb² + yb²)

I have the same problem. Where did a & b come from? Which two vectors are we taking the dot product of? And how is this less expensive?

In the decision function of an SVM, you compute the scalar products of the support vectors (points that are on the margin of your hyperplane, or more precisely, the points that constrain your hyperplane) and your new sample point:

  x· sv
The "z" the article defines is a new component that will be taken into account in the scalar product. A more mathematical way of seeing that is that you define a function phi that takes an original sample of your dataset, and transform it into a new vector. In our case, we simply add a new dimension (x3) based on the two original dimensions (x1, x2) that we add as a third component in our vector:

  phi(x) = [x1, x2, x1² + x2²]
The scalar product we will have to compute in our decision function can then be expressed as (this is the a and b in the article, i.e. the sample and the support vector in our new space):

  phi(x)· phi(sv)
The SVM doesn't need phi(x) or phi(sv), but the scalar product of those two numbers. The kernel trick is to find a function k that satisfies

  k(x, sv) = phi(x)· phi(sv)
and that satisfies the Mercer's condition (I'll let Google explain what it is).

Your SVM will compute this (simpler) k function, instead of the full scalar product. There are multiple "common" kernel functions used (Wikipedia has examples of them[1]), and choosing one is a parameter of your model (ideally, you would then setup a testing protocol to find the best one).

[1] https://en.wikipedia.org/wiki/Positive-definite_kernel#Examp...

Re: An Introduction to Support Vector Machines

#33

Post by sjg007 »

currymj wrote: post #22

Since neural nets are winning at the moment, it's easy to see SVMs as an underdog, being ignored due to deep learning hype and PR. This is kind of true, but it's worth noting that 10-15 years ago we had the exact opposite situation. Neural nets were a once promising technique that had stagnated/hit their limits, while SVMs were the new state of the art. People were coming up with dozens of unnecessary variations on t…

err.. I think it is an evolution. Deep architectures allow for a more efficient function approximation from fewer examples than shallow architectures do.

Deep (neural) networks are really just a generalization of machine learning (on graphs). The key is that we learn the similarity function to discriminate from examples instead of specifying it a priori. You can also build classifiers other ways: linear/logistic regression based on feature vectors or by providing some similarity metric (SVM). But in these cases you are providing the discriminator function.

For example, in SVMs you have to provide a similarity measure as the "kernel" maps your feature vector into a higher dimensional space where the examples can be separated.

In deep neural networks we don't really know (or care to some extent) what the optimal feature vectors are or what the correct similarity metric is. We only care that at the end we've encoded it correctly (e.g. having chosen enough parameters/layers etc...) after training. Again the NN is some general function that applies a soft-max relationship from the inputs to outputs for each layer.

Yann LeCun has a great paper (2007) explaining this:

http://yann.lecun.com/exdb/publis/pdf/bengio-lecun-07.pdf.

Re: An Introduction to Support Vector Machines

#34

Post by nerdponx »

Longwelwind wrote: post #32
tgeery wrote: post #20

Earlier quoted context omitted.

I have the same problem. Where did a & b come from? Which two vectors are we taking the dot product of? And how is this less expensive?

In the decision function of an SVM, you compute the scalar products of the support vectors (points that are on the margin of your hyperplane, or more precisely, the points that constrain your hyperplane) and your new sample point: x· sv The "z" the article defines is a new component that will be taken into account in the scalar product. A more mathematical way of seeing that is that you define a function phi that tak…

Thanks for bringing some saner notation in here. I feel like blog posts and journal articles that abuse notation like this one just make people more allergic to math.

Re: An Introduction to Support Vector Machines

#35

Post by IanCal »

sivvy wrote: post #26
deepnotderp wrote: post #6

I'd just like to note that instead of creating additional animosity between SVMs and deep nets, you could use both together. SVMs with hinge loss can be Yet-another-layer (tm) in your deep net, to be used when it provides better performance.

Could you explain in a bit more detail how you would integrate an SVM layer into a DNN? The kernel matrix depends on all samples, while at training time you would only have access to those in the minibatch.

The simplest is to pop it on the top. Run you DNN to reduce your input down to a nicer cleaner smaller dimensional output, then plop an SVM on top for classification.

Re: An Introduction to Support Vector Machines

#36

Post by nkozyra »

currymj wrote: post #22

Since neural nets are winning at the moment, it's easy to see SVMs as an underdog, being ignored due to deep learning hype and PR. This is kind of true, but it's worth noting that 10-15 years ago we had the exact opposite situation. Neural nets were a once promising technique that had stagnated/hit their limits, while SVMs were the new state of the art. People were coming up with dozens of unnecessary variations on t…

Hell, you can (efficiently) solve a lot of problems with KNN that people are throwing dedicated hardware + Tensorflow. But that's not cool. I've learned that doing the "cool stuff" in the face of practicality is rampant because it's part of the self-fulfilling cycle of hiring people who do the cool stuff and people doing the cool stuff to get hired.

There's a whole world beyond neural networks and it seems like it's mostly all on the backburner now. Which I understand, the resurgence in NN algorithms, approaches and hardware in the last 10 years has been exciting but it does feel like tunnel vision sometimes.

Re: An Introduction to Support Vector Machines

#37

Post by yamaneko »

idrism wrote: post #15

For anyone interested in SVMs (and other introductory Machine Learning concepts), Udacity's intro course is really good: https://www.udacity.com/course/intro-to-machine-learning--ud...

This class from MIT taught by Patrick Winston is also a great resource: https://www.youtube.com/watch?v=_PwhiWxHK8o

At the end of the class, he also gives some historical perspectives, like how Vapnik came up with SVMs.

Re: An Introduction to Support Vector Machines

#38

Post by sivvy »

IanCal wrote: post #35
sivvy wrote: post #26

Earlier quoted context omitted.

Could you explain in a bit more detail how you would integrate an SVM layer into a DNN? The kernel matrix depends on all samples, while at training time you would only have access to those in the minibatch.

The simplest is to pop it on the top. Run you DNN to reduce your input down to a nicer cleaner smaller dimensional output, then plop an SVM on top for classification.

Seems like in that case you would train both models separately on different cost functions. By phrasing it as a layer I was expecting both the SVM and the DNN could be trained simultaneously.

Re: An Introduction to Support Vector Machines

#39

Post by closed »

rcar wrote: post #28
tnecniv wrote: post #27

Earlier quoted context omitted.

What prevented SVMs from catching on in industry?

They tend to create difficult to interpret models that don't perform as well as other "black box" modeling methods (GBMs, neural nets, etc.)

Was this true, or perceived as true in 2003? My understanding was that people did not see them as performing worse than NN back then.

Re: An Introduction to Support Vector Machines

#40

Post by kefka »

thearn4 wrote: post #31
kefka wrote: post #29

Earlier quoted context omitted.

> I wish machine learning research didn't respond so strongly to trends and hype, It's really because nobody actually understands what's going on inside a ML algorithm. When you give it a ginormous dataset, what data is it really using to make its determination of [0.0000999192346 , .91128756789 , 0 , .62819364 , 32.8172] Because what I do for ML is do a supervised fit, then use a next to test and confirm fitness, th…

Shouldn't it be possible to backpropogate those categorical outputs all the way back to the inputs/features (NOT weights) after a forward pass, to localize the sensitivity of them with respect to the actual pixels for a prediction? I imagine that would have to give at least some insight. Beyond that, the convolution/max pool repeated steps could be understood to be applying something akin to a multi-level wavelet dec…

I think you might be on to something, but the big problem here is that the Input is hundreds of GB or TB's . It's hard to understand what a feature is, or even why it's selected.

I can certainly observe what's being selected once the state machine is generated, but I have no clue how it was constructed to make the features. Do determine that, I have to watch the state of the machine as it "grows" to the final result.

Post reply on HN