Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Computer scientists prove why bigger neural networks do better

quantamagazine.org

Post reply on HN
Print view View original

41–50 of 151 posts

Re: Computer scientists prove why bigger neural networks do better

#41

Post by joebob42 »

rackjack wrote: post #34

Silly thought: if bigger NN's are better, shouldn't more neurons be better? Why aren't elephants smarter than us, despite having more neurons? https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n... https://pubmed.ncbi.nlm.nih.gov/24971054/

Aside from other points, more neurons might be better "all else equal", but there are differences between our brain and an elephant's beyond just neuron count.

It's like how just getting a bigger faster computer can help with your problem, but its less powerful than a new more efficient algorithm on the same computer.

Re: Computer scientists prove why bigger neural networks do better

#42

Post by World_Peace »

rackjack wrote: post #34

Silly thought: if bigger NN's are better, shouldn't more neurons be better? Why aren't elephants smarter than us, despite having more neurons? https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n... https://pubmed.ncbi.nlm.nih.gov/24971054/

Elephants very likely could be more intelligent than us, it just seems that intelligence is a difficult thing to measure quantitatively.

Re: Computer scientists prove why bigger neural networks do better

#43

Post by WithinReason »

dan-robertson wrote: post #30
bick_nyers wrote: post #22

Earlier quoted context omitted.

My initial intuition is telling me that it would be diameter/2, from the perspective of a single point, the closest points would be near zero distance away, and the furthest points would be on the opposite side, a full diameter away, and I am assuming that there are a lot of points in a uniform distribution. What I have just thought about though, is what points would be exactly diameter/2 distance away from that poin…

If you consider a point on the sphere it means choosing a bunch of xi such that: x1^2 + x2^2 + … + xn^2 = 1. Suppose wlog you pick (1,0,0,…,0). Then the distance from your point to a random point is: D = (x1-1)^2 + x2^2 + … + xn^2 And from the first equation we know: x1^2 = 1 - x2^2 - x3^2 - … - xn^2 Intuitionistically, your point will be far from a random point if x1 is close to zero, and x1 will be close to zero be…

Let me hijack your explanation starting from this point:

    D = (x₁-1)² + (x₂² + … + xₙ²)
Since all the xₙ² sum to 1, as the dimensionality grows (∑xₙ²→1 as n→∞) each individual xₙ will converge towards 0. Since x₁ is almost 0, therefore the (x₁-1)² term will be almost 1.

Since we know that ∑xₙ²=1, and that x₁² is almost 0, then we also know that ∑xₙ² - x₁² is almost 1, which is the 2nd half of the above expression for D. So the average distance converges to "almost 1 + almost 1", which "almost 2", which is the diameter.

Re: Computer scientists prove why bigger neural networks do better

#44

Post by nazgul17 »

AmericanBlarney wrote: post #32

This conclusion feels like saying more CPU and memory are better. Seems obvious that more moves allows matching to have more nuance, but I guess cool that someone proved it.

From what I understand, it says that more parameters are good. This wasn't obvious before this paper: you can fit a polynomial instead of a neutral net, but adding parameters wouldn't help with robustness in that case: the polynomial would become more and more jagged.

Re: Computer scientists prove why bigger neural networks do better

#45

Post by bo1024 »

leto_ii wrote: post #20
prideout wrote: post #5

> The proof relies on a curious fact about high-dimensional geometry, which is that randomly distributed points placed on the surface of a sphere are almost all a full diameter away from each other. What theorem is this referring to? Sounds like something I should already be familiar with, but I'm not.

I think it's something related to the curse of dimensionality [1] [2], basically just a property of high dimensional spaces (perhaps only certain kinds of spaces though). [1] https://en.wikipedia.org/wiki/Curse_of_dimensionality [2] http://kops.uni-konstanz.de/bitstream/handle/123456789/5715/...

Yes.

Even though almost every all pairs of points are almost a full diameter away from each other, they are also almost all almost orthogonal (i.e. the angle they make with the center of the sphere is very close to 90 degrees).

Re: Computer scientists prove why bigger neural networks do better

#46

Post by molticrystal »

alexpotato wrote: post #40
rackjack wrote: post #34

Silly thought: if bigger NN's are better, shouldn't more neurons be better? Why aren't elephants smarter than us, despite having more neurons? https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n... https://pubmed.ncbi.nlm.nih.gov/24971054/

Because, IIRC, a lot of neurons are dedicated to motion/sensing. Bigger animals may require more neurons to handle moving larger and/or more complicated muscle groups. Interesting related point there is the encephalization quotient which is related to the predicted ratio of brain size to body mass. On the wikipedia page [0] they list the EQ for various animals. Humans are the highest but dolphins and ravens are not f…

To further emphasize that having neural material focused on the appropriate functions is more important vs how much you have, here is a story about a guy whose brain is mostly hollow and filled with fluid, it probably did cause his IQ to be 75 and causes him weakness in his legs, but otherwise he lives a normal adult life more or less.

https://www.newscientist.com/article/dn12301-man-with-tiny-b...

Re: Computer scientists prove why bigger neural networks do better

#47

Post by btilly »

rackjack wrote: post #34

Silly thought: if bigger NN's are better, shouldn't more neurons be better? Why aren't elephants smarter than us, despite having more neurons? https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n... https://pubmed.ncbi.nlm.nih.gov/24971054/

How do you measure intelligence? Elephants have much better memories than we do!

https://www.scientificamerican.com/article/elephants-never-f...

Re: Computer scientists prove why bigger neural networks do better

#48

Post by juancn »

rackjack wrote: post #34

Silly thought: if bigger NN's are better, shouldn't more neurons be better? Why aren't elephants smarter than us, despite having more neurons? https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n... https://pubmed.ncbi.nlm.nih.gov/24971054/

Well, biological neurons are much more complex than CS neurons (https://www.quantamagazine.org/how-computationally-complex-i...).

Also, you're working under the assumption that they are equivalent between mammals which as far as we can tell it's not the case (https://www.medicalnewstoday.com/articles/why-are-human-neur...).

So my guess is that the comparison is much more complex than just number of neurons.

Re: Computer scientists prove why bigger neural networks do better

#49

Post by bee_rider »

World_Peace wrote: post #42
rackjack wrote: post #34

Silly thought: if bigger NN's are better, shouldn't more neurons be better? Why aren't elephants smarter than us, despite having more neurons? https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n... https://pubmed.ncbi.nlm.nih.gov/24971054/

Elephants very likely could be more intelligent than us, it just seems that intelligence is a difficult thing to measure quantitatively.

In particular, a given elephant might be "more intelligent" than a human -- we just happen to have evolved from a particular niche that has rendered us bizarrely good at abstracting knowledge and combining it with the knowledge of other humans.

Re: Computer scientists prove why bigger neural networks do better

#50

Post by peterburkimsher »

gfody wrote: post #38
rackjack wrote: post #34

Silly thought: if bigger NN's are better, shouldn't more neurons be better? Why aren't elephants smarter than us, despite having more neurons? https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n... https://pubmed.ncbi.nlm.nih.gov/24971054/

are we certain they're not? i'm not sure we know how to measure smartness

Dumbo is smarto!

Elephants have bodies built like a tank (and used as such by Hannibal), but humans have better I/O ports.

{reading, writing, listening, speaking, singing, typing, doing, going}

Without opposable thumbs, an elephant is probably quite envious of human writing & typing. Let's use the privilege wisely to encourage one another, teach and learn from each other, from Donald Tusk, and give a helping hand.

Post reply on HN