Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Idiot’s guide to Support vector machines (2003) [pdf]

web.mit.edu

Post reply on HN
Print view View original

11–20 of 64 posts

Re: An Idiot’s guide to Support vector machines (2003) [pdf]

#11

Post by wpietri »

quantombone wrote: post #5

Back in around 2008, SVMs were all the rage in computer vision. We would use hand designed visual features and then a linear SVM on top. That was how object detectors were built (remember DPM?) Funny how SVMs are just max-margin loss functions and we just took for granted that you needed domain expertise to craft features like HOG/SIFT by hand. By 2018, we use ConvNets to learn BOTH the features and the classifier. I…

Could you say a little more about this?

I ask because when we're training human to understand things, there are a variety of benefits to separate feature-understanding from the classifiers. In particular, you get gains in flexibility, extendability, and debuggability.

I get why people are happy to take the ConvNet gains and run with them for now. But have you seen any interesting work to get the benefits of separation in the new paradigm? (Or, alternately, is there a reason why those concerns are outmoded?)

Re: An Idiot’s guide to Support vector machines (2003) [pdf]

#13

Post by soraki_soladead »

wpietri wrote: post #11
quantombone wrote: post #5

Back in around 2008, SVMs were all the rage in computer vision. We would use hand designed visual features and then a linear SVM on top. That was how object detectors were built (remember DPM?) Funny how SVMs are just max-margin loss functions and we just took for granted that you needed domain expertise to craft features like HOG/SIFT by hand. By 2018, we use ConvNets to learn BOTH the features and the classifier. I…

Could you say a little more about this? I ask because when we're training human to understand things, there are a variety of benefits to separate feature-understanding from the classifiers. In particular, you get gains in flexibility, extendability, and debuggability. I get why people are happy to take the ConvNet gains and run with them for now. But have you seen any interesting work to get the benefits of separatio…

That's actually closer to how deep learning started. Initially, deep learning mostly consisted of unsupervised (task independent) features with a linear classifier on top. We had to fit an unsupervised model (e.g. autoencoder) layer by layer before using the feature layers in a supervised task.

This was because we didn't understand how to train a deep model end-to-end until later. When we learned how to make that end-to-end training work it tended to perform better because the learned features were task specific.

You can still learn general features in a bunch of ways, in addition to the older method using autoencoders. For one example, multiple supervised heads with auxiliary losses can learn more generalize features.

Re: An Idiot’s guide to Support vector machines (2003) [pdf]

#14

Post by soraki_soladead »

rdtsc wrote: post #6
rusbus wrote: post #3

It's interesting how quickly support vector machines went from the hot new thing to classify images to an afterthought after deep learning started having great results.

Noticed that too. It feels it was just a few years and all of the sudden everything is "deep" now. The same thing happened with data storage. As soon as big data appeared everyone stopped doing just data and started doing "big data". Now the term is kind of a joke even. I predict in a few years "deep learning" term will become mostly used in an ironic sense as well.

That outcome doesn't seem terribly likely. It's true that, like big data, deep learning is often misused. This is largely because it works well enough in the off-the-shelf case and it's "easier" due to tooling, transfer learning, and free educational materials for beginners. However, deep learning also obtains state-of-the-art in a number of tasks and domains when you know what you're doing.

I don't think your scenario is likely to occur unless something else starts outperforming deep learning (in the broadest sense) _and_ there's an approachable alternative to solve the same problems at least as well.

Re: An Idiot’s guide to Support vector machines (2003) [pdf]

#15

Post by MAXPOOL »

quantombone wrote: post #5

Back in around 2008, SVMs were all the rage in computer vision. We would use hand designed visual features and then a linear SVM on top. That was how object detectors were built (remember DPM?) Funny how SVMs are just max-margin loss functions and we just took for granted that you needed domain expertise to craft features like HOG/SIFT by hand. By 2018, we use ConvNets to learn BOTH the features and the classifier. I…

You still may want to replace softmax layer with support vector machine for classification sometimes.

Re: An Idiot’s guide to Support vector machines (2003) [pdf]

#17

Post by soraki_soladead »

username223 wrote: post #8
rdtsc wrote: post #6

Earlier quoted context omitted.

Noticed that too. It feels it was just a few years and all of the sudden everything is "deep" now. The same thing happened with data storage. As soon as big data appeared everyone stopped doing just data and started doing "big data". Now the term is kind of a joke even. I predict in a few years "deep learning" term will become mostly used in an ironic sense as well.

> I predict in a few years "deep learning" term will become mostly used in an ironic sense as well. I may be a bit behind the times, but I'm also mystified by "deep learning's" popularity. Both giant neural nets and kernel methods have overfitting problems: torture a billion-parameter model long enough, and it will tell you what you want to hear. SVMs address this by finding a large margin for error, which will hopef…

My other comment addresses some of this but you're overstating things a bit. Throwing more data at the model is one solution. Its just not the only, or even best, approach. Properly measured performance on good holdouts and the application of regularization avoids the worst of overfitting. This is standard practice is most of machine learning, not just deep learning.

Deep learning gets a lot of hype because for many applications they perform better and scale better without a lot of tricks and extensions which are now possible with SVMs. You can even use a large margin loss with deep models to get some of the benefits of SVMs.

Adversarial examples are way overblown. First, SVMs are not immune to them either. Second, very few applications are threatened by things like adversarial examples.

Re: An Idiot’s guide to Support vector machines (2003) [pdf]

#18

Post by autokad »

rusbus wrote: post #3

It's interesting how quickly support vector machines went from the hot new thing to classify images to an afterthought after deep learning started having great results.

they were also all the rage in pretty much everything else as well. In the problems not pushed out by DNN, gradient boosting has pretty much replaced SVMs as GBMs are faster training and better accuracy.

Re: An Idiot’s guide to Support vector machines (2003) [pdf]

#19

Post by a-dub »

quantombone wrote: post #5

Back in around 2008, SVMs were all the rage in computer vision. We would use hand designed visual features and then a linear SVM on top. That was how object detectors were built (remember DPM?) Funny how SVMs are just max-margin loss functions and we just took for granted that you needed domain expertise to craft features like HOG/SIFT by hand. By 2018, we use ConvNets to learn BOTH the features and the classifier. I…

So the pitch is that you don't have to do feature engineering... but then instead it seems people do network structure engineering with featurish things like convolutions.

The performance is still better in most cases but I often have to wonder, are people just doing feature engineering once removed and is the better performance just the result of having WAY more parameters in the model?

Re: An Idiot’s guide to Support vector machines (2003) [pdf]

#20

Post by im3w1l »

quantombone wrote: post #5

Back in around 2008, SVMs were all the rage in computer vision. We would use hand designed visual features and then a linear SVM on top. That was how object detectors were built (remember DPM?) Funny how SVMs are just max-margin loss functions and we just took for granted that you needed domain expertise to craft features like HOG/SIFT by hand. By 2018, we use ConvNets to learn BOTH the features and the classifier. I…

To be fair there is a lot of domain knowledge embedded in the use of a convolutional architecture. There is a fascinating paper where the authors don't even train the weights of the convolutional layers and are still able to achieve good performance.

https://arxiv.org/pdf/1606.04801v2.pdf

Post reply on HN