11–20 of 60 posts
Earlier quoted context omitted.
This is not necessarily a trivial fact, but I wouldn't call it incredible. It says a net trained with gradient descent can fit the input perfectly if its much larger than the input. But this says nothing about generalization, i.e. performance on test set -- which is what we really want.
Importantly not SGD, just GD.
Although I'm no expert, isn't this result an incredibly important contribution? This paper claims to prove that: > The current paper proves gradient descent achieves zero training loss in polynomial time for a deep over-parameterized neural network with residual connections (ResNet). If this variant of gradient descent is able to reach global minima in polynomial time, and if neural networks are proven to approximate…
I interpret it as saying: "non-convexity ain't no big deal in this context", and this is the latest contribution in a line of other "no spurious local minima" machine learning theory research. It's interesting for showing that it applies to a rather complex model like ResNet.
1. It's theoretically impossible to guarantee a convergence to global optima using gradient descent if the function is non-convex.
2. The only way to guarantee is to start the gradient descent from different points in the search space or try with different step sizes if the algorithm only starts from the same point in the search space.
3. Also does "achieving zero training loss" mean the network has converged to the global optima? I used to know you will get zero training loss even if you are at a local minima as well.
Please correct me if I am wrong.
I did not understand the paper very well. 1. It's theoretically impossible to guarantee a convergence to global optima using gradient descent if the function is non-convex. 2. The only way to guarantee is to start the gradient descent from different points in the search space or try with different step sizes if the algorithm only starts from the same point in the search space. 3. Also does "achieving zero training lo…
2) This is "a way" not "The only way".
(If A then B) does not imply (if not A then not B)
I did not understand the paper very well. 1. It's theoretically impossible to guarantee a convergence to global optima using gradient descent if the function is non-convex. 2. The only way to guarantee is to start the gradient descent from different points in the search space or try with different step sizes if the algorithm only starts from the same point in the search space. 3. Also does "achieving zero training lo…
Re 2: No.
Re 3: Yes.
I did not understand the paper very well. 1. It's theoretically impossible to guarantee a convergence to global optima using gradient descent if the function is non-convex. 2. The only way to guarantee is to start the gradient descent from different points in the search space or try with different step sizes if the algorithm only starts from the same point in the search space. 3. Also does "achieving zero training lo…
1) Just because you can prove convergence for convex functions does not mean you can't prove it for non-convex functions. 2) This is "a way" not "The only way". (If A then B) does not imply (if not A then not B)
I don't understand. How do you prove a gradient descent is guaranteed to escape local minima?
Earlier quoted context omitted.
1) Just because you can prove convergence for convex functions does not mean you can't prove it for non-convex functions. 2) This is "a way" not "The only way". (If A then B) does not imply (if not A then not B)
>Just because you can prove convergence for convex functions does not mean you can't prove it for non-convex functions. I don't understand. How do you prove a gradient descent is guaranteed to escape local minima?
Although I'm no expert, isn't this result an incredibly important contribution? This paper claims to prove that: > The current paper proves gradient descent achieves zero training loss in polynomial time for a deep over-parameterized neural network with residual connections (ResNet). If this variant of gradient descent is able to reach global minima in polynomial time, and if neural networks are proven to approximate…
Well, without reading the whole paper, two important things strike me. 1. Zero training loss is impossible in most networks because the last layer can only reach the targets asymptotically. 2. Zero training loss means nothing from a practical standpoint. We've had algorithms capable of this for a long time (knn [k=1], decision trees etc.).
2. You clearly have no idea what you are talking about. This paper is trying to argue a bit about why neural networks generalize well by showing with math that a nn with some of their conditions converges to the zero training loss. It isn't remotely meant to be practical. IT IS A THEORETICAL PAPER.
And comparing it to nearest neighbors of 1 is so so so so so silly it isn't even wrong.
edit. #1 is actually an entire research direction in the theory of machine learning fyi.
It is possible to get neural networks that massively overfit but still generalize (which Is weird).
https://arxiv.org/pdf/1611.03530.pdf
That paper was really famous. It showed you can get zero training loss on data when you replace the labels with random noise.
edit 2: I am sorry to be harsh. It is just hard to read such arrant nonsense.
Earlier quoted context omitted.
>Just because you can prove convergence for convex functions does not mean you can't prove it for non-convex functions. I don't understand. How do you prove a gradient descent is guaranteed to escape local minima?
Convex functions aren't the only functions with a single local (and global) minimum - consider sqrt(|x|) for a simple 1d example.