An Introduction to Support Vector Machines • Hacker News Board

Re: An Introduction to Support Vector Machines

#21

Post by genericpseudo » Thu, Jun 22, 2017, 6:43 PM UTC

joe_the_user wrote: post #18

Asdfbla wrote: post #14

I remember that only a few years ago, in a computational statistics class I took the lecturer mentioned how SVMs (and Random Forests) have largely replaced neural networks. How things can change so quickly... I always liked SVMs for the elegance of the kernel trick, but I guess choosing the right kernel functions and parameters for them wasn't that much easier than training a neural net either.

(All this from my rough, amateur understanding), SVMs are more or less equivalent to linear regression in a "feature space" and also equivalent to shallow neural network (~2-3). This means their size more or less increases with the amount of data they are attempting to approximate. And this means they don't do well scaling to truly huge data sets. Deep nets pulled ahead of SVMs at the point people figured out how to…

Close but not quite. The difference between (soft) SVM and a kernel linear classifier is choice of loss function; SVM minimizes hinge loss, linear regression minimizes squared loss.

(Choice of different loss functions will also give you Elastic Net, LASSO, logistic regression. From an engineering point of view I tend to think of the entire class as being different flavors of "stochastic gradient descent", in the spirit of Vowpal Wabbit etc.)

Re: An Introduction to Support Vector Machines

#22

Post by currymj » Thu, Jun 22, 2017, 7:09 PM UTC

Since neural nets are winning at the moment, it's easy to see SVMs as an underdog, being ignored due to deep learning hype and PR. This is kind of true, but it's worth noting that 10-15 years ago we had the exact opposite situation. Neural nets were a once promising technique that had stagnated/hit their limits, while SVMs were the new state of the art.

People were coming up with dozens of unnecessary variations on them, everybody in the world was trying to shoehorn the word "kernel" into their paper titles, using some kind of kernel method was a surefire way to get published.

I wish machine learning research didn't respond so strongly to trends and hype, and I also wish the economics of academic research didn't force people into cliques fighting over scarce resources.

I'm still wondering what, if anything, is going to supplant deep learning. It's probably an existing technique that will suddenly become much more usable due to some small improvement.

Re: An Introduction to Support Vector Machines

#23

Post by Longwelwind » Thu, Jun 22, 2017, 7:10 PM UTC

chrischen wrote: post #9

Can someone explain this part: Imagine the new space we want: z = x² + y² Figure out what the dot product in that space looks like: a · b = xa · xb + ya · yb + za · zb a · b = xa · xb + ya · yb + (xa² + ya²) · (xb² + yb²)

I don't know if you are more familiar with matrix notation

  phi(x) = [x1, x2, x1² + x2²]
  phi(x)· phi(y) = [x1, x2, x1² + x2²]*[y1, y2, y1² + y2²]
  phi(x)· phi(y) = x1 y1 + x2 y2 + (x1² + x2²)(y1² + y²)

Re: An Introduction to Support Vector Machines

#24

Post by rm999 » Thu, Jun 22, 2017, 7:22 PM UTC

currymj wrote: post #22

Since neural nets are winning at the moment, it's easy to see SVMs as an underdog, being ignored due to deep learning hype and PR. This is kind of true, but it's worth noting that 10-15 years ago we had the exact opposite situation. Neural nets were a once promising technique that had stagnated/hit their limits, while SVMs were the new state of the art. People were coming up with dozens of unnecessary variations on t…

This is true, but only in the academic research world. SVMs had relatively little success on practical problems and in industry, so they never built up the kind of standing that neural networks did. Even in 2003-2005 - arguably the peak time for SVMs - neural networks were much better known to almost everyone (industry practitioners, researchers, and laypeople) than SVMs.

What frustrates me is that people who are starting out in machine learning often never learn that linear/logistic regression dominates the practical applications of ML. I've spoken to people who know the in-and-outs of various deep network architectures who don't even know how to start with building a baseline logistic regression model.

Re: An Introduction to Support Vector Machines

#25

Post by timemachiner » Thu, Jun 22, 2017, 7:27 PM UTC

currymj wrote: post #22

Since neural nets are winning at the moment, it's easy to see SVMs as an underdog, being ignored due to deep learning hype and PR. This is kind of true, but it's worth noting that 10-15 years ago we had the exact opposite situation. Neural nets were a once promising technique that had stagnated/hit their limits, while SVMs were the new state of the art. People were coming up with dozens of unnecessary variations on t…

Fashion statements come and go in academia. There are academics doing good research in unfashionable areas and doing just fine. Plus, it's hard to see where the latest trends will go. Who thought Software Defined Networks would be a thing in the 1980s?

Re: An Introduction to Support Vector Machines

#26

Post by sivvy » Thu, Jun 22, 2017, 7:49 PM UTC

deepnotderp wrote: post #6

I'd just like to note that instead of creating additional animosity between SVMs and deep nets, you could use both together. SVMs with hinge loss can be Yet-another-layer (tm) in your deep net, to be used when it provides better performance.

Could you explain in a bit more detail how you would integrate an SVM layer into a DNN? The kernel matrix depends on all samples, while at training time you would only have access to those in the minibatch.

Re: An Introduction to Support Vector Machines

#27

Post by tnecniv » Thu, Jun 22, 2017, 7:54 PM UTC

rm999 wrote: post #24

currymj wrote: post #22

Since neural nets are winning at the moment, it's easy to see SVMs as an underdog, being ignored due to deep learning hype and PR. This is kind of true, but it's worth noting that 10-15 years ago we had the exact opposite situation. Neural nets were a once promising technique that had stagnated/hit their limits, while SVMs were the new state of the art. People were coming up with dozens of unnecessary variations on t…

This is true, but only in the academic research world. SVMs had relatively little success on practical problems and in industry, so they never built up the kind of standing that neural networks did. Even in 2003-2005 - arguably the peak time for SVMs - neural networks were much better known to almost everyone (industry practitioners, researchers, and laypeople) than SVMs. What frustrates me is that people who are sta…

What prevented SVMs from catching on in industry?

Re: An Introduction to Support Vector Machines

#28

Post by rcar » Thu, Jun 22, 2017, 7:59 PM UTC

tnecniv wrote: post #27

rm999 wrote: post #24

Earlier quoted context omitted.

This is true, but only in the academic research world. SVMs had relatively little success on practical problems and in industry, so they never built up the kind of standing that neural networks did. Even in 2003-2005 - arguably the peak time for SVMs - neural networks were much better known to almost everyone (industry practitioners, researchers, and laypeople) than SVMs. What frustrates me is that people who are sta…

What prevented SVMs from catching on in industry?

They tend to create difficult to interpret models that don't perform as well as other "black box" modeling methods (GBMs, neural nets, etc.)

Re: An Introduction to Support Vector Machines

#29

Post by kefka » Thu, Jun 22, 2017, 8:03 PM UTC

currymj wrote: post #22

Since neural nets are winning at the moment, it's easy to see SVMs as an underdog, being ignored due to deep learning hype and PR. This is kind of true, but it's worth noting that 10-15 years ago we had the exact opposite situation. Neural nets were a once promising technique that had stagnated/hit their limits, while SVMs were the new state of the art. People were coming up with dozens of unnecessary variations on t…

> I wish machine learning research didn't respond so strongly to trends and hype,

It's really because nobody actually understands what's going on inside a ML algorithm. When you give it a ginormous dataset, what data is it really using to make its determination of

[0.0000999192346 , .91128756789 , 0 , .62819364 , 32.8172]

Because what I do for ML is do a supervised fit, then use a next to test and confirm fitness, then unleash it on untrained data and check and see. But I have no real understanding of what those numbers actually represent. I mean, does .91128756789 represent the curve around the nose, or is it skin color, or is it a facial encoding of 3d shape?

> I'm still wondering what, if anything, is going to supplant deep learning.

I think it'll be a slow climb to actual understanding. Right now, we have object identifiers in NN. They work, after TB's of images and PFLOPS of cpu/gpu time. It's only brute force with 'magic black boxes' - and that provides results but no understanding. The next steps are actually deciphering what the understanding is, or making straight-up algorithms that can differentiate between things.

Re: An Introduction to Support Vector Machines

#30

Post by rm999 » Thu, Jun 22, 2017, 8:14 PM UTC

tnecniv wrote: post #27

rm999 wrote: post #24

Earlier quoted context omitted.

This is true, but only in the academic research world. SVMs had relatively little success on practical problems and in industry, so they never built up the kind of standing that neural networks did. Even in 2003-2005 - arguably the peak time for SVMs - neural networks were much better known to almost everyone (industry practitioners, researchers, and laypeople) than SVMs. What frustrates me is that people who are sta…

What prevented SVMs from catching on in industry?

the tldr answer: SVMs are theoretically powerful but either impractical or pointless.

The longer answer:

SVMs comes with two theoretical benefits: 1. A guaranteed optimal solution. You get this with simpler techniques like logistic regression. 2. The ability to use non-linear kernels, which can offer much more power than logistic regression (on par with neural networks).

So, at first, it seemed like SVMs were the best of all worlds, and a lot of people got excited. In practice, though, non-linear kernels slowed training down to the point of being impractical. Linear kernels were fast enough, but removed the second benefit, so most people would prefer to use established linear techniques like logistic regression.