Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Computer scientists prove why bigger neural networks do better

quantamagazine.org

Post reply on HN
Print view View original

51–60 of 151 posts

Re: Computer scientists prove why bigger neural networks do better

#51

Post by stevofolife »

qorrect wrote: post #16
feketegy wrote: post #2

This looks interesting, I bookmarked it. My biggest blocker is the "statistics" part of M/L, knowing what algorithms to choose for various cases.

This book was a big help for me and is very well written, https://xcelab.net/rm/statistical-rethinking/ . You can find it free online ( along with video course ). The printed version is a very nice high quality book.

Do you know if there are any online classes that use this book as a reference? Or more generally, what type of courses teaches this subject?

Re: Computer scientists prove why bigger neural networks do better

#53

Post by notahacker »

bee_rider wrote: post #49
World_Peace wrote: post #42

Earlier quoted context omitted.

Elephants very likely could be more intelligent than us, it just seems that intelligence is a difficult thing to measure quantitatively.

In particular, a given elephant might be "more intelligent" than a human -- we just happen to have evolved from a particular niche that has rendered us bizarrely good at abstracting knowledge and combining it with the knowledge of other humans.

What is "more intelligent" if not "more capable of abstracting, synthesizing and sharing knowledge"?

Re: Computer scientists prove why bigger neural networks do better

#54

Post by beebeepka »

gfody wrote: post #38
rackjack wrote: post #34

Silly thought: if bigger NN's are better, shouldn't more neurons be better? Why aren't elephants smarter than us, despite having more neurons? https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n... https://pubmed.ncbi.nlm.nih.gov/24971054/

are we certain they're not? i'm not sure we know how to measure smartness

I only stopped saying my cat is smarter than the vast majority of people I've met because she is no longer with us.

I did, and still do, believe this to be true. Would love to befriend a bird

Re: Computer scientists prove why bigger neural networks do better

#55

Post by jayd16 »

notahacker wrote: post #53
bee_rider wrote: post #49

Earlier quoted context omitted.

In particular, a given elephant might be "more intelligent" than a human -- we just happen to have evolved from a particular niche that has rendered us bizarrely good at abstracting knowledge and combining it with the knowledge of other humans.

What is "more intelligent" if not "more capable of abstracting, synthesizing and sharing knowledge"?

You'd also need the desire for such things.

Re: Computer scientists prove why bigger neural networks do better

#56

Post by Retric »

dan-robertson wrote: post #35
Retric wrote: post #15

Earlier quoted context omitted.

It’s just wrong as stated, there is only one point a full diameter away from each point on a high dimensional sphere. Aka (1,0,0,0,0, …) maps to (-1,0,0,0,0, …) and nothing else. Just as (1,0) maps to (-1,0) on a unit circle and (1,0,0) maps to (-1,0,0) on a unit sphere. On a high dimensional sphere they should generally be close to square root of 2 radius away from each other.

The fact should say that the expected distance between two random points tends to the diameter as the dimension increases. The intuition is that to be close you need to be close in a large number of coordinates and the law of large numbers (though coordinates aren’t independent) suggests that is unlikely. If you fix one point on a sphere (say (1,0,…,0)) then, for a high dimension, most points will not have any extrem…

You forgot the square root on distance formula. Distance = square root of ((X1 - X2) ^ 2 + (Y1 - Y2) ^2 + …).

Consider the origin (0,0,0, …) to a random point on the sphere (~0, ~0, ~0, …). So Distance from origin = square root of ((~0-0)^2 + (~0-0)^2 + (~0-0)^2 + … ), which sums to 1 by definition of the unit high dimensional sphere.

Then plug in 1 vs 0 in the first place because we care about (1,0,0,0 …) and you get the correct answer = square root of ((~0-1)^2 + (~0-0)^2 + (~0-0)^2 + … ) ~= square root of 2.

Edited to fix typo and add clarity.

Re: Computer scientists prove why bigger neural networks do better

#57

Post by acchow »

molticrystal wrote: post #46
alexpotato wrote: post #40

Earlier quoted context omitted.

Because, IIRC, a lot of neurons are dedicated to motion/sensing. Bigger animals may require more neurons to handle moving larger and/or more complicated muscle groups. Interesting related point there is the encephalization quotient which is related to the predicted ratio of brain size to body mass. On the wikipedia page [0] they list the EQ for various animals. Humans are the highest but dolphins and ravens are not f…

To further emphasize that having neural material focused on the appropriate functions is more important vs how much you have, here is a story about a guy whose brain is mostly hollow and filled with fluid, it probably did cause his IQ to be 75 and causes him weakness in his legs, but otherwise he lives a normal adult life more or less. https://www.newscientist.com/article/dn12301-man-with-tiny-b...

Doesn't this demonstrate the opposite of what you were claiming?

Re: Computer scientists prove why bigger neural networks do better

#58

Post by ska »

AmericanBlarney wrote: post #32

This conclusion feels like saying more CPU and memory are better. Seems obvious that more moves allows matching to have more nuance, but I guess cool that someone proved it.

> Seems obvious that more moves allows matching to have more nuance,

This really has to be balance against overfitting. The key problem in ML is generalization, and lots of things improve training performance while making that worse.

Re: Computer scientists prove why bigger neural networks do better

#59

Post by lacksconfidence »

acchow wrote: post #57
molticrystal wrote: post #46

Earlier quoted context omitted.

To further emphasize that having neural material focused on the appropriate functions is more important vs how much you have, here is a story about a guy whose brain is mostly hollow and filled with fluid, it probably did cause his IQ to be 75 and causes him weakness in his legs, but otherwise he lives a normal adult life more or less. https://www.newscientist.com/article/dn12301-man-with-tiny-b...

Doesn't this demonstrate the opposite of what you were claiming?

I feel like the quotes agree with parent:

> “If something happens very slowly over quite some time, maybe over decades, the different parts of the brain take up functions that would normally be done by the part that is pushed to the side,” adds Muenke, who was not involved in the case.

Re: Computer scientists prove why bigger neural networks do better

#60

Post by Ajedi32 »

acchow wrote: post #57
molticrystal wrote: post #46

Earlier quoted context omitted.

To further emphasize that having neural material focused on the appropriate functions is more important vs how much you have, here is a story about a guy whose brain is mostly hollow and filled with fluid, it probably did cause his IQ to be 75 and causes him weakness in his legs, but otherwise he lives a normal adult life more or less. https://www.newscientist.com/article/dn12301-man-with-tiny-b...

Doesn't this demonstrate the opposite of what you were claiming?

Did you see the scans? The dude's head is practically empty (brain 55-75% smaller than normal) and nobody even noticed until he was 44 years old and got an MRI.
Post reply on HN