Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Idiot’s guide to Support vector machines (2003) [pdf]

web.mit.edu

Post reply on HN
Print view View original

51–60 of 64 posts

Re: An Idiot’s guide to Support vector machines (2003) [pdf]

#51

Post by srean »

cultus wrote: post #23

I notice this doesn't mention hinge loss, which is by far the simpler way of arriving at the SVM. Hinge loss is just max(0, 1- t*y), where y is the output of the linear model and t = +-1 is the label. Thus, it takes the common-sense approach of not penalizing losses that are far enough away from the decision boundary, and penalizing linearly after that. An SVM is literally just a linear model with hinge loss instead…

Its similar to L2regularized logistic regression not the pure logistic regression. Although in your defense pure logistic is not used that often. I find the large margin view very intuitive -- one finds a separator such that the labeled data points are separated and lie as far as possible from the line

Re: An Idiot’s guide to Support vector machines (2003) [pdf]

#52

Post by srean »

olooney wrote: post #45
iamwil wrote: post #33

I have a question! In the pdf, it said that the optimization problem in SVMs have a nice property in that it was quadratic, which means that there's a nice global minimum to go towards, and not lots of local minimum like in NN. That means, it seems SVMs won't get stuck at a suboptimal solution. Is that not a problem in DNNs now? Or is it that it's such high dimensionality that local minima don't stop the optimizer, b…

People who know more about deep learning than I do tend to argue that there is empirical evidence that non-convexity is a non-issue because the performance of local minima will be close to the performance of the global minimum, given a sufficient number of nodes[1][2]. One such quote: I once ran a small neural net 100 times on simple three-dimensional data re- selecting the initial weights to be small and random on e…

Thanks for your point about BLUE and regularized solutions. I had not considered that point of view before

Re: An Idiot’s guide to Support vector machines (2003) [pdf]

#53

Post by cultus »

srean wrote: post #51
cultus wrote: post #23

I notice this doesn't mention hinge loss, which is by far the simpler way of arriving at the SVM. Hinge loss is just max(0, 1- t*y), where y is the output of the linear model and t = +-1 is the label. Thus, it takes the common-sense approach of not penalizing losses that are far enough away from the decision boundary, and penalizing linearly after that. An SVM is literally just a linear model with hinge loss instead…

Its similar to L2regularized logistic regression not the pure logistic regression. Although in your defense pure logistic is not used that often. I find the large margin view very intuitive -- one finds a separator such that the labeled data points are separated and lie as far as possible from the line

Logistic regression is usually assumed to be regularized in machine learning. It's not often you find practical problems that don't need regularization.

Re: An Idiot’s guide to Support vector machines (2003) [pdf]

#54

Post by srean »

cultus wrote: post #53
srean wrote: post #51

Earlier quoted context omitted.

Its similar to L2regularized logistic regression not the pure logistic regression. Although in your defense pure logistic is not used that often. I find the large margin view very intuitive -- one finds a separator such that the labeled data points are separated and lie as far as possible from the line

Logistic regression is usually assumed to be regularized in machine learning. It's not often you find practical problems that don't need regularization.

Indeed. You would notice that I had said as much. For the analogy between SVMs and LR to work you need L2 squared regression specifically, other regularizers, for example L1 wont work for the analogy. L2 squared is the key for the connection with Hilbert space and kernels to fall out automagically

Re: An Idiot’s guide to Support vector machines (2003) [pdf]

#55

Post by currymj »

iamwil wrote: post #33

I have a question! In the pdf, it said that the optimization problem in SVMs have a nice property in that it was quadratic, which means that there's a nice global minimum to go towards, and not lots of local minimum like in NN. That means, it seems SVMs won't get stuck at a suboptimal solution. Is that not a problem in DNNs now? Or is it that it's such high dimensionality that local minima don't stop the optimizer, b…

As others have said, you don't actually want the global optimum of a neural network because that would be terrible overfitting. There is some evidence that architectural tricks (like ResNet) that empirically help performance are making the loss landscape "more convex", though.

https://arxiv.org/pdf/1712.09913.pdf

Re: An Idiot’s guide to Support vector machines (2003) [pdf]

#56

Post by abhgh »

nl wrote: post #46
a-dub wrote: post #19

Earlier quoted context omitted.

So the pitch is that you don't have to do feature engineering... but then instead it seems people do network structure engineering with featurish things like convolutions. The performance is still better in most cases but I often have to wonder, are people just doing feature engineering once removed and is the better performance just the result of having WAY more parameters in the model?

are people just doing feature engineering once removed and is the better performance just the result of having WAY more parameters in the model? Not really, or sort of, depending on how you think. A deep neural network does work - at least to some extent - because of the large number of parameters. However, it is practical because it can be trained in a reasonable amount of time. Things like ResNets are useful becaus…

But you won't be able to train it to the same accuracy.

I'm not sure I agree with this bit in theory. A Neural Network is a stack of basis functions; and this stack can also be seen as a bunch of basis functions. And basis functions are what kernels represent. Trivially, you could then "copy" the weights that a ANN would learn into a kernel and obtain the same accuracy.

The reason this doesn't work in practice is, in SVMs, you tend not to learn kernels from scratch but use (possibly a combination of) standard parameterized kernels - [1]. The learning step in the SVM adapts this standard kernel to your dataset as much as the parameters allow, but this would be sub-optimal compared to learning a kernel (or the corresponding basis functions) from scratch that's built just for your data. With a well trained ANN the latter is what you get.

[1] there has been a fair amount of work on learning kernels too, but its not as mainstream as using standard kernels.

Re: An Idiot’s guide to Support vector machines (2003) [pdf]

#57

Post by abhgh »

rpedela wrote: post #36
abhgh wrote: post #12

If you need closer to a ELI5 version I recommend this - [1]. Disclaimer: written by me. [1] https://blog.statsbot.co/support-vector-machines-tutorial-c1...

Definitely a better fit for idiots like me. BTW that is a compliment. It is very hard to make something complicated easy to understand. You succeeded.

Thanks!

Re: An Idiot’s guide to Support vector machines (2003) [pdf]

#58

Post by abhgh »

jameslk wrote: post #40
abhgh wrote: post #12

If you need closer to a ELI5 version I recommend this - [1]. Disclaimer: written by me. [1] https://blog.statsbot.co/support-vector-machines-tutorial-c1...

Great article! I did not understand the part with kernels before, so I appreciate the simplified explanation. The interactive demo on https://www.csie.ntu.edu.tw/~cjlin/libsvm/ is really cool.

Thanks! Yes its a pretty good demo, it should be more popular IMO.

Re: An Idiot’s guide to Support vector machines (2003) [pdf]

#60

Post by abhgh »

olooney wrote: post #45
iamwil wrote: post #33

I have a question! In the pdf, it said that the optimization problem in SVMs have a nice property in that it was quadratic, which means that there's a nice global minimum to go towards, and not lots of local minimum like in NN. That means, it seems SVMs won't get stuck at a suboptimal solution. Is that not a problem in DNNs now? Or is it that it's such high dimensionality that local minima don't stop the optimizer, b…

People who know more about deep learning than I do tend to argue that there is empirical evidence that non-convexity is a non-issue because the performance of local minima will be close to the performance of the global minimum, given a sufficient number of nodes[1][2]. One such quote: I once ran a small neural net 100 times on simple three-dimensional data re- selecting the initial weights to be small and random on e…

I think you're exactly right about the modularity aspect of DL; in fact I made a similar comment on this page, albeit speaking in terms of basis functions.

I have a minor nitpick regarding this point you make: not that anything is likely to beat RBF. Depending on the data, specialized kernels can help immensely. An easy example is sequence classification where something like a string kernel might work really well. Or image classification, where histogram based kernels might prove superior.

Note that sometimes you might want to measure how good a kernel is for a problem not by its prediction accuracy alone but also by the number of support vectors it needs - if the final model retains ~100% of the training data as support vectors, it is not a great model in some (subjective) sense since it is memorizing a lot. Depending on the data, you might "beat" the RBF kernel on this aspect too.

Regarding the training time, there are some interesting tricks I've come across (but not tried them out yet) -[1], [2].

[1] Ensemble SVMs http://www.jmlr.org/papers/volume15/claesen14a/claesen14a.pd...

[2] SVMPath - algorithm to fit the entire path of SVM solutions for every value of the cost parameter, with essentially the same computational cost as fitting one SVM model. http://www.jmlr.org/papers/volume5/hastie04a/hastie04a.pdf

Post reply on HN