An Idiot’s guide to Support vector machines (2003) [pdf] • Hacker News Board

Re: An Idiot’s guide to Support vector machines (2003) [pdf]

#61

Post by nl » Tue, Jan 01, 2019, 11:26 AM UTC

abhgh wrote: post #56

nl wrote: post #46

Earlier quoted context omitted.

are people just doing feature engineering once removed and is the better performance just the result of having WAY more parameters in the model? Not really, or sort of, depending on how you think. A deep neural network does work - at least to some extent - because of the large number of parameters. However, it is practical because it can be trained in a reasonable amount of time. Things like ResNets are useful becaus…

But you won't be able to train it to the same accuracy. I'm not sure I agree with this bit in theory . A Neural Network is a stack of basis functions; and this stack can also be seen as a bunch of basis functions. And basis functions are what kernels represent. Trivially, you could then "copy" the weights that a ANN would learn into a kernel and obtain the same accuracy. The reason this doesn't work in practice is, i…

Trivially, you could then "copy" the weights that a ANN would learn into a kernel and obtain the same accuracy.

Sure.

I'm not sure I agree with this bit in theory.

No one really agrees with it in theory - I'm not aware of a good theoretical explanation as to why some deep networks are easier to train. And yet there is a growing body of real, generalized practical hints which work pretty reliably.

This is pretty exciting! There is undiscovered ideas here. But it is unsatisfactory from the theoretical sense at the moment.

Re: An Idiot’s guide to Support vector machines (2003) [pdf]

#62

Post by igorkraw » Tue, Jan 01, 2019, 4:59 PM UTC

nightski wrote: post #31

dplavery92 wrote: post #29

Earlier quoted context omitted.

Significantly, it becomes a simple, 2-layer neural network. The power of the advances of neural networks in the past decade have largely relied on "deep" architectures with many layers. Very deep networks effectively learn the features from the data, rather than learn a decision surface over a set of hand-crafted features, as in learning with SVMs or shallow neural networks.

I thought it had been proven that a two layer neural network has the same power as a deep one (obviously with a much greater width). It's just that deep neural networks are a lot more practical to train in practice. So I'm not sure how important that distinction is.

If i remember my Goodfellow correctly (and quickly checking, wikipedia, I did https://en.wikipedia.org/wiki/Universal_approximation_theore... ), there is a nuance here which is almost always missed: you can represent any function with a sufficiently wide 2 layer neural network, it doesn't say anything about being able tune the network until you find a correct setting (i.e. learnability).

This is important. Flippantly said,discarding learnability and speed of convergence, you can get the power of any neural network by the following algorithm:

1. Randomly generates a sufficiently wide bit pattern 2. Interprets it as a program and run it on the test set 3. discard results until the desired accuracy is reached

Re: An Idiot’s guide to Support vector machines (2003) [pdf]

#63

Post by igorkraw » Tue, Jan 01, 2019, 5:00 PM UTC

yters wrote: post #35

dplavery92 wrote: post #34

Earlier quoted context omitted.

This is something of an academic factoid that has nothing to do with the practice of training and using neural networks, or with the merits of deep networks that I was describing above. Shallow feed-forward networks are "universal function approximators" [0] when the number of hidden neurons is finite but unbounded. Of course, the width of that layer grows exponentially in the depth of the deep network that you might…

If an SVM kernel can replicate a 2 layer NN, why couldn't there be a kernel for a X layer NN, and then autoderive the architecture just like SVMs can autoderive the correct number of neurons? Then there'd also be a more robust theoretical understanding of what's happening.

See my other point, there might be, in fact there definitely is for any working NN, but as of now (2019, happy new year) we probably can't find it

Re: An Idiot’s guide to Support vector machines (2003) [pdf]

#64

Post by igorkraw » Tue, Jan 01, 2019, 5:01 PM UTC

dplavery92 wrote: post #34

nightski wrote: post #31

Earlier quoted context omitted.

I thought it had been proven that a two layer neural network has the same power as a deep one (obviously with a much greater width). It's just that deep neural networks are a lot more practical to train in practice. So I'm not sure how important that distinction is.

This is something of an academic factoid that has nothing to do with the practice of training and using neural networks, or with the merits of deep networks that I was describing above. Shallow feed-forward networks are "universal function approximators" [0] when the number of hidden neurons is finite but unbounded. Of course, the width of that layer grows exponentially in the depth of the deep network that you might…

An important addition to this is priors: deep networks allow to express the prior that hierarchical representation, i.e. composing into multiple layers of abstractions make sense (see e.g. conv nets).