Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Introduction to Support Vector Machines

monkeylearn.com

Post reply on HN
Print view View original

41–50 of 64 posts

Re: An Introduction to Support Vector Machines

#41

Post by rm999 »

closed wrote: post #39
rcar wrote: post #28

Earlier quoted context omitted.

They tend to create difficult to interpret models that don't perform as well as other "black box" modeling methods (GBMs, neural nets, etc.)

Was this true, or perceived as true in 2003? My understanding was that people did not see them as performing worse than NN back then.

Definitely true. I worked for a company that was generating millions of dollars a year from neural networks in the mid 90s (edit: to be clear I didn't work there in the 90s, I joined years after their initial buildouts). The Unreasonable Effectiveness™ of neural networks has been true for a long time. When I worked there I tried switching out some models with SVMs and they were less accurate and took 1-2 orders of magnitude more time to train.

Re: An Introduction to Support Vector Machines

#42

Post by shas3 »

tnecniv wrote: post #27
rm999 wrote: post #24

Earlier quoted context omitted.

This is true, but only in the academic research world. SVMs had relatively little success on practical problems and in industry, so they never built up the kind of standing that neural networks did. Even in 2003-2005 - arguably the peak time for SVMs - neural networks were much better known to almost everyone (industry practitioners, researchers, and laypeople) than SVMs. What frustrates me is that people who are sta…

What prevented SVMs from catching on in industry?

Also, hyperparameter optimization is a bit painful. Zoubin Ghahramani of Cambridge is a champion for Gaussian process models which are as flexible as SVM/SVC, but have more disciplined approach: via well-chosen priors (Bayesian structure) and single-step hyperparameter optimization.

Re: An Introduction to Support Vector Machines

#43

Post by azeirah »

thearn4 wrote: post #31
kefka wrote: post #29

Earlier quoted context omitted.

> I wish machine learning research didn't respond so strongly to trends and hype, It's really because nobody actually understands what's going on inside a ML algorithm. When you give it a ginormous dataset, what data is it really using to make its determination of [0.0000999192346 , .91128756789 , 0 , .62819364 , 32.8172] Because what I do for ML is do a supervised fit, then use a next to test and confirm fitness, th…

Shouldn't it be possible to backpropogate those categorical outputs all the way back to the inputs/features (NOT weights) after a forward pass, to localize the sensitivity of them with respect to the actual pixels for a prediction? I imagine that would have to give at least some insight. Beyond that, the convolution/max pool repeated steps could be understood to be applying something akin to a multi-level wavelet dec…

I'm in no way a researcher or even an enthusiast of machine learning, but I'm pretty sure that I came across a paper posted on HN a few days ago that did exactly what you and the parent poster are describing, figuring out what pixels contributed most to some machine learning algorithm. I'll try and see if I can find it.

Edit: yep, found it.

SmoothGrad: removing noise by adding noise, https://arxiv.org/abs/1706.03825

Web page with explanations and examples

https://tensorflow.github.io/saliency/

I couldn't find the HN thread, but there was no discussion as far as I remember.

Re: An Introduction to Support Vector Machines

#44

Post by YZF »

rm999 wrote: post #24
currymj wrote: post #22

Since neural nets are winning at the moment, it's easy to see SVMs as an underdog, being ignored due to deep learning hype and PR. This is kind of true, but it's worth noting that 10-15 years ago we had the exact opposite situation. Neural nets were a once promising technique that had stagnated/hit their limits, while SVMs were the new state of the art. People were coming up with dozens of unnecessary variations on t…

This is true, but only in the academic research world. SVMs had relatively little success on practical problems and in industry, so they never built up the kind of standing that neural networks did. Even in 2003-2005 - arguably the peak time for SVMs - neural networks were much better known to almost everyone (industry practitioners, researchers, and laypeople) than SVMs. What frustrates me is that people who are sta…

Weren't SVMs used for the NetFlix recommendation engine? (i.e. the NetFlix prize?)

Re: An Introduction to Support Vector Machines

#45

Post by [deleted] »

azeirah wrote: post #43
thearn4 wrote: post #31

Earlier quoted context omitted.

Shouldn't it be possible to backpropogate those categorical outputs all the way back to the inputs/features (NOT weights) after a forward pass, to localize the sensitivity of them with respect to the actual pixels for a prediction? I imagine that would have to give at least some insight. Beyond that, the convolution/max pool repeated steps could be understood to be applying something akin to a multi-level wavelet dec…

I'm in no way a researcher or even an enthusiast of machine learning, but I'm pretty sure that I came across a paper posted on HN a few days ago that did exactly what you and the parent poster are describing, figuring out what pixels contributed most to some machine learning algorithm. I'll try and see if I can find it. Edit: yep, found it. SmoothGrad: removing noise by adding noise, https://arxiv.org/abs/1706.03825…

[deleted]

Re: An Introduction to Support Vector Machines

#46

Post by denzil_correa »

YZF wrote: post #44
rm999 wrote: post #24

Earlier quoted context omitted.

This is true, but only in the academic research world. SVMs had relatively little success on practical problems and in industry, so they never built up the kind of standing that neural networks did. Even in 2003-2005 - arguably the peak time for SVMs - neural networks were much better known to almost everyone (industry practitioners, researchers, and laypeople) than SVMs. What frustrates me is that people who are sta…

Weren't SVMs used for the NetFlix recommendation engine? (i.e. the NetFlix prize?)

I think it was Gradient Decision Boosting Trees to combine different models.

Re: An Introduction to Support Vector Machines

#47

Post by tgeery »

Longwelwind wrote: post #32
tgeery wrote: post #20

Earlier quoted context omitted.

I have the same problem. Where did a & b come from? Which two vectors are we taking the dot product of? And how is this less expensive?

In the decision function of an SVM, you compute the scalar products of the support vectors (points that are on the margin of your hyperplane, or more precisely, the points that constrain your hyperplane) and your new sample point: x· sv The "z" the article defines is a new component that will be taken into account in the scalar product. A more mathematical way of seeing that is that you define a function phi that tak…

Thank you. This was an amazing explanation. I am new to SVM's but did not make the connection that margin points (observations along the margin of the hyperplane) become your support vectors. This makes a lot more sense.

And if I am following correctly, it would make sense that the final step would then be:

We would maximize the dot product of a new observation with the support vectors to determine its classification (red or blue)

Re: An Introduction to Support Vector Machines

#48

Post by closed »

rm999 wrote: post #41
closed wrote: post #39

Earlier quoted context omitted.

Was this true, or perceived as true in 2003? My understanding was that people did not see them as performing worse than NN back then.

Definitely true. I worked for a company that was generating millions of dollars a year from neural networks in the mid 90s (edit: to be clear I didn't work there in the 90s, I joined years after their initial buildouts). The Unreasonable Effectiveness™ of neural networks has been true for a long time. When I worked there I tried switching out some models with SVMs and they were less accurate and took 1-2 orders of ma…

Really useful to hear, thanks! I know psychology was gaining a lot of headway with NN models in the 90s, but had little sense for what was going on in industry.

Re: An Introduction to Support Vector Machines

#49

Post by eggie5 »

YZF wrote: post #44
rm999 wrote: post #24

Earlier quoted context omitted.

This is true, but only in the academic research world. SVMs had relatively little success on practical problems and in industry, so they never built up the kind of standing that neural networks did. Even in 2003-2005 - arguably the peak time for SVMs - neural networks were much better known to almost everyone (industry practitioners, researchers, and laypeople) than SVMs. What frustrates me is that people who are sta…

Weren't SVMs used for the NetFlix recommendation engine? (i.e. the NetFlix prize?)

No, it was an ensemble of Collaborative filtering using Matrix Factorization (using SVD) and a RBM.

Re: An Introduction to Support Vector Machines

#50

Post by tanilama »

tnecniv wrote: post #27
rm999 wrote: post #24

Earlier quoted context omitted.

This is true, but only in the academic research world. SVMs had relatively little success on practical problems and in industry, so they never built up the kind of standing that neural networks did. Even in 2003-2005 - arguably the peak time for SVMs - neural networks were much better known to almost everyone (industry practitioners, researchers, and laypeople) than SVMs. What frustrates me is that people who are sta…

What prevented SVMs from catching on in industry?

1.SVM is not interpretable just as DL.

2.Hard to parallel if you're using kernels other than linear one.

3.So-so performance.

Post reply on HN