Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Computer scientists prove why bigger neural networks do better

quantamagazine.org

Post reply on HN
Print view View original

31–40 of 151 posts

Re: Computer scientists prove why bigger neural networks do better

#31

Post by pfortuny »

aix1 wrote: post #11
prideout wrote: post #5

> The proof relies on a curious fact about high-dimensional geometry, which is that randomly distributed points placed on the surface of a sphere are almost all a full diameter away from each other. What theorem is this referring to? Sounds like something I should already be familiar with, but I'm not.

Not my area of expertise, but the quoted "fact" seems at best incompletely stated: surely for it to hold there must be some constraints on the number of points (likely as a function of the diameter)?

It is for VERY HUGE n, as siblings explain.

Re: Computer scientists prove why bigger neural networks do better

#33

Post by woopwoop »

zwaps wrote: post #18

Can someone speak to the generality of assuming c-isoperimetry for the distribution of features? Without knowing anything about this in particular, this seems to be a rather pertinent restriction of the result related to things like sampling assumptions and the like.

It really depends on what we assume about "natural" data. If it looks "positively curved", e.g. the uniform measure on the boundary of a convex body, or the a gaussian, or something, this holds. But if the distribution exhibits a strong hierarchical structure that's not so good. I think it's a plausible if not obviously true assumption.

Re: Computer scientists prove why bigger neural networks do better

#35

Post by dan-robertson »

Retric wrote: post #15
aix1 wrote: post #11

Earlier quoted context omitted.

Not my area of expertise, but the quoted "fact" seems at best incompletely stated: surely for it to hold there must be some constraints on the number of points (likely as a function of the diameter)?

It’s just wrong as stated, there is only one point a full diameter away from each point on a high dimensional sphere. Aka (1,0,0,0,0, …) maps to (-1,0,0,0,0, …) and nothing else. Just as (1,0) maps to (-1,0) on a unit circle and (1,0,0) maps to (-1,0,0) on a unit sphere. On a high dimensional sphere they should generally be close to square root of 2 radius away from each other.

The fact should say that the expected distance between two random points tends to the diameter as the dimension increases. The intuition is that to be close you need to be close in a large number of coordinates and the law of large numbers (though coordinates aren’t independent) suggests that is unlikely. If you fix one point on a sphere (say (1,0,…,0)) then, for a high dimension, most points will not have any extreme values in coordinates and will look like (~0,~0,…,~0) where ~0 means something close to zero. But if we sum the squares of everything apart from the first we get 1 - (~0)^2 ~= 1, so the distance from our fixed point is (1 - ~0)^2 + sum_2^n (0 - ~0)^2 ~= 1 + 1 = 2.

Re: Computer scientists prove why bigger neural networks do better

#36

Post by kemiller »

rackjack wrote: post #34

Silly thought: if bigger NN's are better, shouldn't more neurons be better? Why aren't elephants smarter than us, despite having more neurons? https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n... https://pubmed.ncbi.nlm.nih.gov/24971054/

IANANS but my understanding is that neurons/body mass is more indicative. Large animals have more neurons because large bodies need more.

Re: Computer scientists prove why bigger neural networks do better

#37

Post by cloogshicer »

rackjack wrote: post #34

Silly thought: if bigger NN's are better, shouldn't more neurons be better? Why aren't elephants smarter than us, despite having more neurons? https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n... https://pubmed.ncbi.nlm.nih.gov/24971054/

You probably already know this (since you wrote "silly thought"), but real-life neurons are ridiculously more complex than simulated "neurons" in an NN. So the analogy doesn't really hold.

Re: Computer scientists prove why bigger neural networks do better

#38

Post by gfody »

rackjack wrote: post #34

Silly thought: if bigger NN's are better, shouldn't more neurons be better? Why aren't elephants smarter than us, despite having more neurons? https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n... https://pubmed.ncbi.nlm.nih.gov/24971054/

are we certain they're not? i'm not sure we know how to measure smartness

Re: Computer scientists prove why bigger neural networks do better

#39

Post by moyix »

aray wrote: post #14
SpaceManNabs wrote: post #3

> Right now, we are routinely creating neural networks that have a number of parameters more than the number of training samples. This says that the books have to be rewritten. Confused by this statement. Double descent with overparameterization is exhibited in "classical settings" too and mentioned in older books. > In their new proof, the pair show that overparameterization is necessary for a network to be robust.…

> Double descent with overparameterization is exhibited in "classical settings" too and mentioned in older books. I’m curious for references or citations to this. When I was going over double descent I tried to find citations like this (just in a couple places like ML/stats textbooks).

Here's one that lists some older references: https://arxiv.org/abs/2004.04328

Re: Computer scientists prove why bigger neural networks do better

#40

Post by alexpotato »

rackjack wrote: post #34

Silly thought: if bigger NN's are better, shouldn't more neurons be better? Why aren't elephants smarter than us, despite having more neurons? https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n... https://pubmed.ncbi.nlm.nih.gov/24971054/

Because, IIRC, a lot of neurons are dedicated to motion/sensing.

Bigger animals may require more neurons to handle moving larger and/or more complicated muscle groups.

Interesting related point there is the encephalization quotient which is related to the predicted ratio of brain size to body mass. On the wikipedia page [0] they list the EQ for various animals. Humans are the highest but dolphins and ravens are not far behind.

Post reply on HN