Gradient Descent Finds Global Minima of Deep Neural Networks
1–10 of 60 posts
Re: Gradient Descent Finds Global Minima of Deep Neural Networks
#2I cant claim to fully understand the proof, but these guys have done an amazing job in terms of furthering our understanding of deep nets.
Re: Gradient Descent Finds Global Minima of Deep Neural Networks
#3Although I'm no expert, isn't this result an incredibly important contribution? This paper claims to prove that:
> The current paper proves gradient descent achieves zero training loss in polynomial time for a deep over-parameterized neural network with residual connections (ResNet).
If this variant of gradient descent is able to reach global minima in polynomial time, and if neural networks are proven to approximate any function, then ostensibly this technique could be used to guarantee the lowest error possible in approximating any function. This seems incredibly important. Can someone correct my reading of the abstract?
Re: Gradient Descent Finds Global Minima of Deep Neural Networks
#4Although I'm no expert, isn't this result an incredibly important contribution? This paper claims to prove that: > The current paper proves gradient descent achieves zero training loss in polynomial time for a deep over-parameterized neural network with residual connections (ResNet). If this variant of gradient descent is able to reach global minima in polynomial time, and if neural networks are proven to approximate…
The paper claims to reach the global minima of a given neural network in polynomial time. The time complexity of constructing a neural network that approximates any function is an entirely different matter. I’m not even sure how one would begin to approximate a highly algorithmic process (e.g., a hash function) using a neural network.
Re: Gradient Descent Finds Global Minima of Deep Neural Networks
#5Although I'm no expert, isn't this result an incredibly important contribution? This paper claims to prove that: > The current paper proves gradient descent achieves zero training loss in polynomial time for a deep over-parameterized neural network with residual connections (ResNet). If this variant of gradient descent is able to reach global minima in polynomial time, and if neural networks are proven to approximate…
Well, without reading the whole paper, two important things strike me.
1. Zero training loss is impossible in most networks because the last layer can only reach the targets asymptotically.
2. Zero training loss means nothing from a practical standpoint. We've had algorithms capable of this for a long time (knn [k=1], decision trees etc.).
Re: Gradient Descent Finds Global Minima of Deep Neural Networks
#6Although I'm no expert, isn't this result an incredibly important contribution? This paper claims to prove that: > The current paper proves gradient descent achieves zero training loss in polynomial time for a deep over-parameterized neural network with residual connections (ResNet). If this variant of gradient descent is able to reach global minima in polynomial time, and if neural networks are proven to approximate…
Well, without reading the whole paper, two important things strike me. 1. Zero training loss is impossible in most networks because the last layer can only reach the targets asymptotically. 2. Zero training loss means nothing from a practical standpoint. We've had algorithms capable of this for a long time (knn [k=1], decision trees etc.).
Yeah. And they require that the number of parameters grows at least polynomially with the size of input. In this regime, we also have another algorithm capable of zero loss -- the linear regression!
Re: Gradient Descent Finds Global Minima of Deep Neural Networks
#7Although I'm no expert, isn't this result an incredibly important contribution? This paper claims to prove that: > The current paper proves gradient descent achieves zero training loss in polynomial time for a deep over-parameterized neural network with residual connections (ResNet). If this variant of gradient descent is able to reach global minima in polynomial time, and if neural networks are proven to approximate…
This is not necessarily a trivial fact, but I wouldn't call it incredible. It says a net trained with gradient descent can fit the input perfectly if its much larger than the input.
But this says nothing about generalization, i.e. performance on test set -- which is what we really want.
Re: Gradient Descent Finds Global Minima of Deep Neural Networks
#8Although I'm no expert, isn't this result an incredibly important contribution? This paper claims to prove that: > The current paper proves gradient descent achieves zero training loss in polynomial time for a deep over-parameterized neural network with residual connections (ResNet). If this variant of gradient descent is able to reach global minima in polynomial time, and if neural networks are proven to approximate…
I interpret it as saying: "non-convexity ain't no big deal in this context", and this is the latest contribution in a line of other "no spurious local minima" machine learning theory research. It's interesting for showing that it applies to a rather complex model like ResNet.
Re: Gradient Descent Finds Global Minima of Deep Neural Networks
#9Although I'm no expert, isn't this result an incredibly important contribution? This paper claims to prove that: > The current paper proves gradient descent achieves zero training loss in polynomial time for a deep over-parameterized neural network with residual connections (ResNet). If this variant of gradient descent is able to reach global minima in polynomial time, and if neural networks are proven to approximate…
This is not necessarily a trivial fact, but I wouldn't call it incredible. It says a net trained with gradient descent can fit the input perfectly if its much larger than the input. But this says nothing about generalization, i.e. performance on test set -- which is what we really want.
Importantly not SGD, just GD.
Re: Gradient Descent Finds Global Minima of Deep Neural Networks
#10My hope is that as these bounds are refined we can start to do BotE calculations such as, "I have 50k training images of 512x512x3 and 1k classes, this means I'll need a Convolutional Resnet of at most 12 layers and 12M params to fit the training data so let's start at half that." Rather than today which is 'let's use resnet 101 and see if that works.'