Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Idiot’s guide to Support vector machines (2003) [pdf]

web.mit.edu

Post reply on HN
Print view View original

21–30 of 64 posts

Re: An Idiot’s guide to Support vector machines (2003) [pdf]

#21

Post by a-dub »

a-dub wrote: post #19
quantombone wrote: post #5

Back in around 2008, SVMs were all the rage in computer vision. We would use hand designed visual features and then a linear SVM on top. That was how object detectors were built (remember DPM?) Funny how SVMs are just max-margin loss functions and we just took for granted that you needed domain expertise to craft features like HOG/SIFT by hand. By 2018, we use ConvNets to learn BOTH the features and the classifier. I…

So the pitch is that you don't have to do feature engineering... but then instead it seems people do network structure engineering with featurish things like convolutions. The performance is still better in most cases but I often have to wonder, are people just doing feature engineering once removed and is the better performance just the result of having WAY more parameters in the model?

I guess one upshot to the SVM approach is that there's math for quantifying how well a given model will generalize, subject to some assumptions.

Is there anything like that in the ANN world?

Re: An Idiot’s guide to Support vector machines (2003) [pdf]

#23

Post by cultus »

I notice this doesn't mention hinge loss, which is by far the simpler way of arriving at the SVM. Hinge loss is just max(0, 1- t*y), where y is the output of the linear model and t = +-1 is the label. Thus, it takes the common-sense approach of not penalizing losses that are far enough away from the decision boundary, and penalizing linearly after that.

An SVM is literally just a linear model with hinge loss instead of log loss (logistic regression) or squared loss (ordinary linear regression) in primal form. For apparently historical reasons, it is usually derived from the "hard-margin" SVM in dual form, motivating with trying to maximize the margin. This is complicated and not very intuitive.

This also causes people to conflate the kernel trick and the dual form, while in fact they have nothing to do with each other. You can use the kernel trick in primal svm just fine.

Stochastic gradient descent can also be used for primal methods, while it doesn't work in the dual. That makes it much faster for large problems than the dual.

Re: An Idiot’s guide to Support vector machines (2003) [pdf]

#24

Post by cscurmudgeon »

quantombone wrote: post #5

Back in around 2008, SVMs were all the rage in computer vision. We would use hand designed visual features and then a linear SVM on top. That was how object detectors were built (remember DPM?) Funny how SVMs are just max-margin loss functions and we just took for granted that you needed domain expertise to craft features like HOG/SIFT by hand. By 2018, we use ConvNets to learn BOTH the features and the classifier. I…

I feel like we pushed features into the architecture and called it a day.

Otherwise, why we would we need a gazillion architectures for different problems (or even the same exact problem)?

Re: An Idiot’s guide to Support vector machines (2003) [pdf]

#25

Post by computerex »

a-dub wrote: post #21
a-dub wrote: post #19

Earlier quoted context omitted.

So the pitch is that you don't have to do feature engineering... but then instead it seems people do network structure engineering with featurish things like convolutions. The performance is still better in most cases but I often have to wonder, are people just doing feature engineering once removed and is the better performance just the result of having WAY more parameters in the model?

I guess one upshot to the SVM approach is that there's math for quantifying how well a given model will generalize, subject to some assumptions. Is there anything like that in the ANN world?

In short, no. Not for practically large models used in common tasks like image classification or speech to text.

Re: An Idiot’s guide to Support vector machines (2003) [pdf]

#27

Post by quantombone »

cultus wrote: post #23

I notice this doesn't mention hinge loss, which is by far the simpler way of arriving at the SVM. Hinge loss is just max(0, 1- t*y), where y is the output of the linear model and t = +-1 is the label. Thus, it takes the common-sense approach of not penalizing losses that are far enough away from the decision boundary, and penalizing linearly after that. An SVM is literally just a linear model with hinge loss instead…

The hinge-loss and the primal form of the SVM objective is really easy to understand. Every ML 101 class would jump into the dual formulation, talk about kernels, RKHS, and all the fancy stuff.

Once you realize that a linear SVM isn’t very different from logistic regression, it starts to all make sense (at least it did for me).

Key insight of the hinge-loss: once something is classified correctly beyond the margin, it incurs a loss of zero.

Now, Something fun to think about. Draw the hinge loss. Now draw the ReLU (which is found all over the place in CNNs). Now thing about L1-regularization (which was used to induce sparsity in compressed sensing). They are more similar in form than you would think.

Re: An Idiot’s guide to Support vector machines (2003) [pdf]

#29

Post by dplavery92 »

yters wrote: post #9
quantombone wrote: post #5

Back in around 2008, SVMs were all the rage in computer vision. We would use hand designed visual features and then a linear SVM on top. That was how object detectors were built (remember DPM?) Funny how SVMs are just max-margin loss functions and we just took for granted that you needed domain expertise to craft features like HOG/SIFT by hand. By 2018, we use ConvNets to learn BOTH the features and the classifier. I…

If you use the right sort of kernel for an SVM it becomes a neural network with automatic architecture derivation. See slide 7: http://www.cs.rpi.edu/~magdon/courses/LFD-Slides/SlidesLect2...

Significantly, it becomes a simple, 2-layer neural network. The power of the advances of neural networks in the past decade have largely relied on "deep" architectures with many layers. Very deep networks effectively learn the features from the data, rather than learn a decision surface over a set of hand-crafted features, as in learning with SVMs or shallow neural networks.

Re: An Idiot’s guide to Support vector machines (2003) [pdf]

#30

Post by simonw »

simonw wrote: post #28

Bullet point on page 2: "Optimal hyperplane for linearly separable patterns" I think the author may be working from a very different definition of the word "idiot".

Seconding the recommendation for https://blog.statsbot.co/support-vector-machines-tutorial-c1... - after reading that, "Optimal hyperplane for linearly separable patterns" actually made sense to me.
Post reply on HN