Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Computer scientists prove why bigger neural networks do better

quantamagazine.org

Post reply on HN
Print view View original

11–20 of 151 posts

Re: Computer scientists prove why bigger neural networks do better

#11

Post by aix1 »

prideout wrote: post #5

> The proof relies on a curious fact about high-dimensional geometry, which is that randomly distributed points placed on the surface of a sphere are almost all a full diameter away from each other. What theorem is this referring to? Sounds like something I should already be familiar with, but I'm not.

Not my area of expertise, but the quoted "fact" seems at best incompletely stated: surely for it to hold there must be some constraints on the number of points (likely as a function of the diameter)?

Re: Computer scientists prove why bigger neural networks do better

#13

Post by renewiltord »

stared wrote: post #8

I am surprised that the paper does not even cite the Lottery Ticket Hypothesis ( https://arxiv.org/abs/1803.03635 , https://eng.uber.com/deconstructing-lottery-tickets/ ). In the LTH paper (IMHO the most fundamental deep learning publication in the last few years), the number of tickets goes as layer_size^n_layers.

Considering the subject, it is at least somewhat amusing that you double posted this.

Re: Computer scientists prove why bigger neural networks do better

#14

Post by aray »

SpaceManNabs wrote: post #3

> Right now, we are routinely creating neural networks that have a number of parameters more than the number of training samples. This says that the books have to be rewritten. Confused by this statement. Double descent with overparameterization is exhibited in "classical settings" too and mentioned in older books. > In their new proof, the pair show that overparameterization is necessary for a network to be robust.…

> Double descent with overparameterization is exhibited in "classical settings" too and mentioned in older books.

I’m curious for references or citations to this. When I was going over double descent I tried to find citations like this (just in a couple places like ML/stats textbooks).

Re: Computer scientists prove why bigger neural networks do better

#15

Post by Retric »

aix1 wrote: post #11
prideout wrote: post #5

> The proof relies on a curious fact about high-dimensional geometry, which is that randomly distributed points placed on the surface of a sphere are almost all a full diameter away from each other. What theorem is this referring to? Sounds like something I should already be familiar with, but I'm not.

Not my area of expertise, but the quoted "fact" seems at best incompletely stated: surely for it to hold there must be some constraints on the number of points (likely as a function of the diameter)?

It’s just wrong as stated, there is only one point a full diameter away from each point on a high dimensional sphere. Aka (1,0,0,0,0, …) maps to (-1,0,0,0,0, …) and nothing else. Just as (1,0) maps to (-1,0) on a unit circle and (1,0,0) maps to (-1,0,0) on a unit sphere.

On a high dimensional sphere they should generally be close to square root of 2 radius away from each other.

Re: Computer scientists prove why bigger neural networks do better

#16

Post by qorrect »

feketegy wrote: post #2

This looks interesting, I bookmarked it. My biggest blocker is the "statistics" part of M/L, knowing what algorithms to choose for various cases.

This book was a big help for me and is very well written, https://xcelab.net/rm/statistical-rethinking/ . You can find it free online ( along with video course ). The printed version is a very nice high quality book.

Re: Computer scientists prove why bigger neural networks do better

#18

Post by zwaps »

Can someone speak to the generality of assuming c-isoperimetry for the distribution of features?

Without knowing anything about this in particular, this seems to be a rather pertinent restriction of the result related to things like sampling assumptions and the like.

Re: Computer scientists prove why bigger neural networks do better

#19

Post by ravi-delia »

aix1 wrote: post #11
prideout wrote: post #5

> The proof relies on a curious fact about high-dimensional geometry, which is that randomly distributed points placed on the surface of a sphere are almost all a full diameter away from each other. What theorem is this referring to? Sounds like something I should already be familiar with, but I'm not.

Not my area of expertise, but the quoted "fact" seems at best incompletely stated: surely for it to hold there must be some constraints on the number of points (likely as a function of the diameter)?

It should be almost all points are almost a full diameter away. However it's still very striking, and an unintuitive fact about very high dimensional spheres.

Re: Computer scientists prove why bigger neural networks do better

#20

Post by leto_ii »

prideout wrote: post #5

> The proof relies on a curious fact about high-dimensional geometry, which is that randomly distributed points placed on the surface of a sphere are almost all a full diameter away from each other. What theorem is this referring to? Sounds like something I should already be familiar with, but I'm not.

I think it's something related to the curse of dimensionality [1] [2], basically just a property of high dimensional spaces (perhaps only certain kinds of spaces though).

[1] https://en.wikipedia.org/wiki/Curse_of_dimensionality

[2] http://kops.uni-konstanz.de/bitstream/handle/123456789/5715/...

Post reply on HN