Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Computer scientists prove why bigger neural networks do better

quantamagazine.org

Post reply on HN
Print view View original

61–70 of 151 posts

Re: Computer scientists prove why bigger neural networks do better

#61

Post by divbzero »

acchow wrote: post #57
molticrystal wrote: post #46

Earlier quoted context omitted.

To further emphasize that having neural material focused on the appropriate functions is more important vs how much you have, here is a story about a guy whose brain is mostly hollow and filled with fluid, it probably did cause his IQ to be 75 and causes him weakness in his legs, but otherwise he lives a normal adult life more or less. https://www.newscientist.com/article/dn12301-man-with-tiny-b...

Doesn't this demonstrate the opposite of what you were claiming?

I think it’s that a priori you would expect a hollow brain to have a far more drastic effect and not allow for a mostly normal adult life.

Re: Computer scientists prove why bigger neural networks do better

#62

Post by Retric »

bick_nyers wrote: post #25
Retric wrote: post #15

Earlier quoted context omitted.

It’s just wrong as stated, there is only one point a full diameter away from each point on a high dimensional sphere. Aka (1,0,0,0,0, …) maps to (-1,0,0,0,0, …) and nothing else. Just as (1,0) maps to (-1,0) on a unit circle and (1,0,0) maps to (-1,0,0) on a unit sphere. On a high dimensional sphere they should generally be close to square root of 2 radius away from each other.

Euclidean distance calculations change based on number of dimensions, for example, in 3 dimensions it is sqrt(a^2+b^2+c^2).

Yes, that’s why it’s square root of 2. Consider the origin (0,0,0, …) to a random point on the sphere (~0, ~0, ~0, …).

Distance = square root of ((X1 - X2) ^ 2 + (Y1 - Y2) ^2 + …). So D = square root of ((~0-0)^2 + (~0-0)^2 + (~0-0)^2 + … ), which is equal to 1 by definition of the unit high dimensional sphere.

So distance from (1,0,0,0 …) to (~0, ~0, ~0, …) = square root of ((~0-1)^2 + (~0-0)^2 + (~0-0)^2 + … ) ~= square root of 2.

Re: Computer scientists prove why bigger neural networks do better

#63

Post by Ajedi32 »

btilly wrote: post #47
rackjack wrote: post #34

Silly thought: if bigger NN's are better, shouldn't more neurons be better? Why aren't elephants smarter than us, despite having more neurons? https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n... https://pubmed.ncbi.nlm.nih.gov/24971054/

How do you measure intelligence? Elephants have much better memories than we do! https://www.scientificamerican.com/article/elephants-never-f...

That article doesn't seem to support your claim. All of the feats mentioned would be entirely unremarkable in your average human.

Re: Computer scientists prove why bigger neural networks do better

#64

Post by wizzwizz4 »

beebeepka wrote: post #54
gfody wrote: post #38

Earlier quoted context omitted.

are we certain they're not? i'm not sure we know how to measure smartness

I only stopped saying my cat is smarter than the vast majority of people I've met because she is no longer with us. I did, and still do, believe this to be true. Would love to befriend a bird

You can befriend corvids. Teaching them symbolic language is tricky, but they can trade and socialise and solve puzzles (if you manage to explain the puzzle).

Re: Computer scientists prove why bigger neural networks do better

#65

Post by gwern »

molticrystal wrote: post #46
alexpotato wrote: post #40

Earlier quoted context omitted.

Because, IIRC, a lot of neurons are dedicated to motion/sensing. Bigger animals may require more neurons to handle moving larger and/or more complicated muscle groups. Interesting related point there is the encephalization quotient which is related to the predicted ratio of brain size to body mass. On the wikipedia page [0] they list the EQ for various animals. Humans are the highest but dolphins and ravens are not f…

To further emphasize that having neural material focused on the appropriate functions is more important vs how much you have, here is a story about a guy whose brain is mostly hollow and filled with fluid, it probably did cause his IQ to be 75 and causes him weakness in his legs, but otherwise he lives a normal adult life more or less. https://www.newscientist.com/article/dn12301-man-with-tiny-b...

Volume != neurons. In any case, 75 is awful and is usually considered borderline retarded. (If you're tempted to respond with other cases of higher IQ, note that they are often retracted or unconfirmed and likely fraudulent in some way; see https://www.gwern.net/Hydrocephalus .)

Re: Computer scientists prove why bigger neural networks do better

#66

Post by pishpash »

divbzero wrote: post #61
acchow wrote: post #57

Earlier quoted context omitted.

Doesn't this demonstrate the opposite of what you were claiming?

I think it’s that a priori you would expect a hollow brain to have a far more drastic effect and not allow for a mostly normal adult life.

Why would you expect that, when a tiny insect can do pretty intelligent things? What "unexpected" things humans can do are probably all in the >75 IQ range.

Re: Computer scientists prove why bigger neural networks do better

#67

Post by gwern »

stared wrote: post #8

I am surprised that the paper does not even cite the Lottery Ticket Hypothesis ( https://arxiv.org/abs/1803.03635 , https://eng.uber.com/deconstructing-lottery-tickets/ ). In the LTH paper (IMHO the most fundamental deep learning publication in the last few years), the number of tickets goes as layer_size^n_layers.

I don't see how lottery tickets yield the isoperimetry result, even in a heuristic or handwavy sort of way. Yes, a larger network is more likely to have good-scoring subnetworks; sure. But that's all it says. What does that tell me about how efficiently I can construct an adversarial example? For that, I need something else, like, say a geometric argument about what sort of network will interpolate between high-dimensional datapoints with properties like "not changing much in response to small input changes"...

Re: Computer scientists prove why bigger neural networks do better

#68

Post by pishpash »

cloogshicer wrote: post #37
rackjack wrote: post #34

Silly thought: if bigger NN's are better, shouldn't more neurons be better? Why aren't elephants smarter than us, despite having more neurons? https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n... https://pubmed.ncbi.nlm.nih.gov/24971054/

You probably already know this (since you wrote "silly thought"), but real-life neurons are ridiculously more complex than simulated "neurons" in an NN. So the analogy doesn't really hold.

They're more complex in biological construction and in signaling mechanism, but no proof that they are more complex in function.

Re: Computer scientists prove why bigger neural networks do better

#69

Post by willmw101 »

gwern wrote: post #65
molticrystal wrote: post #46

Earlier quoted context omitted.

To further emphasize that having neural material focused on the appropriate functions is more important vs how much you have, here is a story about a guy whose brain is mostly hollow and filled with fluid, it probably did cause his IQ to be 75 and causes him weakness in his legs, but otherwise he lives a normal adult life more or less. https://www.newscientist.com/article/dn12301-man-with-tiny-b...

Volume != neurons. In any case, 75 is awful and is usually considered borderline retarded. (If you're tempted to respond with other cases of higher IQ, note that they are often retracted or unconfirmed and likely fraudulent in some way; see https://www.gwern.net/Hydrocephalus .)

>Volume != neurons

Exactly. Most of the newer research on this topic suggests that it's neural connection complexity, and specifically frontal lobe volume, rather than overall brain size that determines intelligence or brain power.

https://neuroscience.stanford.edu/news/ask-neuroscientist-do...

>Luckily, there is much more to a brain when you look at it under a microscope, and most neuroscientists now believe that the complexity of cellular and molecular organization of neural connections, or synapses, is what truly determines a brain’s computational capacity. This view is supported by findings that intelligence is more correlated with frontal lobe volume and volume of gray matter, which is dense in neural cell bodies and synapses, than sheer brain size. Other research comparing proteins at synapses between different species suggests that what makes up synapses at the molecular level has had a huge impact on intelligence throughout evolutionary history. So, although having a big brain is somewhat predictive of having big smarts, intelligence probably depends much more on how efficiently different parts of your brain communicate with each other.

Re: Computer scientists prove why bigger neural networks do better

#70

Post by fabiospampinato »

rackjack wrote: post #34

Silly thought: if bigger NN's are better, shouldn't more neurons be better? Why aren't elephants smarter than us, despite having more neurons? https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n... https://pubmed.ncbi.nlm.nih.gov/24971054/

I read on wikipedia [0] the other day a fairly disturbing statistic related to this, apparently human men have on average a ~10% bigger brain than women. It'd be interesting to know if that translates to a higher neuron count or the difference in volume is due to something else.

[0]: https://en.wikipedia.org/wiki/Brain_size#:~:text=In%20men%20....

Post reply on HN