Computer scientists prove why bigger neural networks do better
quantamagazine.org
Computer scientists prove why bigger neural networks do better
1–10 of 151 posts
Re: Computer scientists prove why bigger neural networks do better
#2This looks interesting, I bookmarked it.
My biggest blocker is the "statistics" part of M/L, knowing what algorithms to choose for various cases.
Re: Computer scientists prove why bigger neural networks do better
#3> Right now, we are routinely creating neural networks that have a number of parameters more than the number of training samples. This says that the books have to be rewritten.
Confused by this statement. Double descent with overparameterization is exhibited in "classical settings" too and mentioned in older books.
> In their new proof, the pair show that overparameterization is necessary for a network to be robust.
What is important to note here is that many of papers this paper cites prove or show this result in certain network architectures. This paper adds universality.
> The proof is very elementary — no heavy math, and it says something very general
The most elementary part was clever use of Hoeffding's inequality. Some people are really fast readers haha.
I don't even know how you pick up the fact that isoperimetry holds in manifold settings with positive curvature while also playing with all those norms and inequalities. A few years ago I mentioned on here all the maths that I knew or wanted to know to read more papers, and others critiqued that the list was too long. Well, this is why!
Re: Computer scientists prove why bigger neural networks do better
#4This looks interesting, I bookmarked it. My biggest blocker is the "statistics" part of M/L, knowing what algorithms to choose for various cases.
check out introduction to statistical learning
Re: Computer scientists prove why bigger neural networks do better
#5> The proof relies on a curious fact about high-dimensional geometry, which is that randomly distributed points placed on the surface of a sphere are almost all a full diameter away from each other.
What theorem is this referring to? Sounds like something I should already be familiar with, but I'm not.
Re: Computer scientists prove why bigger neural networks do better
#6Off topic, but I love that they make it trivial to find a link to the original paper. I know not everyone loves quanta, but stuff like this is really refreshing.
Re: Computer scientists prove why bigger neural networks do better
#7[deleted]
Re: Computer scientists prove why bigger neural networks do better
#8I am surprised that the paper does not even cite the Lottery Ticket Hypothesis (https://arxiv.org/abs/1803.03635, https://eng.uber.com/deconstructing-lottery-tickets/).
In the LTH paper (IMHO the most fundamental deep learning publication in the last few years), the number of tickets goes as layer_size^n_layers.
Re: Computer scientists prove why bigger neural networks do better
#9> The proof relies on a curious fact about high-dimensional geometry, which is that randomly distributed points placed on the surface of a sphere are almost all a full diameter away from each other. What theorem is this referring to? Sounds like something I should already be familiar with, but I'm not.
[deleted]
Re: Computer scientists prove why bigger neural networks do better
#10> The proof relies on a curious fact about high-dimensional geometry, which is that randomly distributed points placed on the surface of a sphere are almost all a full diameter away from each other. What theorem is this referring to? Sounds like something I should already be familiar with, but I'm not.
[deleted]