Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Computer scientists prove why bigger neural networks do better

quantamagazine.org

Post reply on HN
Print view View original

81–90 of 151 posts

Re: Computer scientists prove why bigger neural networks do better

#81

Post by stjohnswarts »

user90349032 wrote: post #79
gwern wrote: post #75

Earlier quoted context omitted.

Density is also important. If we look at other things - some recent studies have been done on number-counting ( https://royalsocietypublishing.org/doi/10.1098/rstb.2020.052... ) or bird brains ( https://www.gwern.net/docs/psychology/neuroscience/2020-herc... ) - density jumps out as a major predictor. African elephants may have some more neurons, but the density isn't as great as a human where it counts, so they are…

And yet, no animal except humans is self aware. Really makes you wonder why that is.

that is simply incorrect bonobos, orcas, elephants, dolphins, chimpanzees, etc have all shown degrees of self awareness.

Re: Computer scientists prove why bigger neural networks do better

#82

Post by moomin »

user90349032 wrote: post #79
gwern wrote: post #75

Earlier quoted context omitted.

Density is also important. If we look at other things - some recent studies have been done on number-counting ( https://royalsocietypublishing.org/doi/10.1098/rstb.2020.052... ) or bird brains ( https://www.gwern.net/docs/psychology/neuroscience/2020-herc... ) - density jumps out as a major predictor. African elephants may have some more neurons, but the density isn't as great as a human where it counts, so they are…

And yet, no animal except humans is self aware. Really makes you wonder why that is.

Probably that you don’t know how to measure what you’re describing.

Plenty of animals recognise themselves in the mirror, for instance.

Re: Computer scientists prove why bigger neural networks do better

#83

Post by lariati »

World_Peace wrote: post #42
rackjack wrote: post #34

Silly thought: if bigger NN's are better, shouldn't more neurons be better? Why aren't elephants smarter than us, despite having more neurons? https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n... https://pubmed.ncbi.nlm.nih.gov/24971054/

Elephants very likely could be more intelligent than us, it just seems that intelligence is a difficult thing to measure quantitatively.

I have listened to François Chollet say that all intelligence is specialized intelligence.

I suspect the question really doesn't make sense if that is true.

We just have this bias/mind projection fallacy that intelligence is a general physical property of the brain that can be measured. I just suspect this is not true.

Like athletic ability doesn't generalize well. Of course, someone not athletic at all is never going to be a great athlete in anything but it makes no sense to compare Lance Armstrong to Patrick Mahomes in some general athletic context. Putting a number on a general athletic ability index between the two would just be total nonsense.

Re: Computer scientists prove why bigger neural networks do better

#84

Post by Swizec »

user90349032 wrote: post #79
gwern wrote: post #75

Earlier quoted context omitted.

Density is also important. If we look at other things - some recent studies have been done on number-counting ( https://royalsocietypublishing.org/doi/10.1098/rstb.2020.052... ) or bird brains ( https://www.gwern.net/docs/psychology/neuroscience/2020-herc... ) - density jumps out as a major predictor. African elephants may have some more neurons, but the density isn't as great as a human where it counts, so they are…

And yet, no animal except humans is self aware. Really makes you wonder why that is.

There are lots of self aware non human animals.

Dolphins and elephants are famous examples, most primates as well. Even many birds show levels of self awareness and theory of mind (they know the difference between what they know and what others know)

Re: Computer scientists prove why bigger neural networks do better

#85

Post by q845712 »

user90349032 wrote: post #79
gwern wrote: post #75

Earlier quoted context omitted.

Density is also important. If we look at other things - some recent studies have been done on number-counting ( https://royalsocietypublishing.org/doi/10.1098/rstb.2020.052... ) or bird brains ( https://www.gwern.net/docs/psychology/neuroscience/2020-herc... ) - density jumps out as a major predictor. African elephants may have some more neurons, but the density isn't as great as a human where it counts, so they are…

And yet, no animal except humans is self aware. Really makes you wonder why that is.

are you sure? https://en.wikipedia.org/wiki/Theory_of_mind_in_animals

Re: Computer scientists prove why bigger neural networks do better

#86

Post by mattkrause »

gwern wrote: post #65
molticrystal wrote: post #46

Earlier quoted context omitted.

To further emphasize that having neural material focused on the appropriate functions is more important vs how much you have, here is a story about a guy whose brain is mostly hollow and filled with fluid, it probably did cause his IQ to be 75 and causes him weakness in his legs, but otherwise he lives a normal adult life more or less. https://www.newscientist.com/article/dn12301-man-with-tiny-b...

Volume != neurons. In any case, 75 is awful and is usually considered borderline retarded. (If you're tempted to respond with other cases of higher IQ, note that they are often retracted or unconfirmed and likely fraudulent in some way; see https://www.gwern.net/Hydrocephalus .)

As a counterpoint, rats without a cortex can do...basically everything normal rats can do--except trim their toenails. The classic reference for this is Whitslaw's 1990 chapter "The decorticate rat".

This thread has links to a copy, plus a bunch of related studies in humans and animals. https://twitter.com/markdhumphries/status/107105276276554137...

Re: Computer scientists prove why bigger neural networks do better

#87

Post by mattkrause »

pishpash wrote: post #68
cloogshicer wrote: post #37

Earlier quoted context omitted.

You probably already know this (since you wrote "silly thought"), but real-life neurons are ridiculously more complex than simulated "neurons" in an NN. So the analogy doesn't really hold.

They're more complex in biological construction and in signaling mechanism, but no proof that they are more complex in function.

An individual biological neuron can compute a variety of functions, including max and xor, that a single perceptron can't (e.g., https://www.science.org/doi/10.1126/science.aax6239 ). In general, one needs a fairly elaborate ANN to approximate the behavior of a single biological neuron.

OTOH, a three-layer network is a universal function approximator and RNNs are universal dynamical systems approximators, so they are sort of trivially equivalent.

Re: Computer scientists prove why bigger neural networks do better

#88

Post by visarga »

Swizec wrote: post #84
user90349032 wrote: post #79

Earlier quoted context omitted.

And yet, no animal except humans is self aware. Really makes you wonder why that is.

There are lots of self aware non human animals. Dolphins and elephants are famous examples, most primates as well. Even many birds show levels of self awareness and theory of mind (they know the difference between what they know and what others know)

Seems like being a social animal is necessary for self awareness.

Re: Computer scientists prove why bigger neural networks do better

#89

Post by bick_nyers »

Retric wrote: post #62
bick_nyers wrote: post #25

Earlier quoted context omitted.

Euclidean distance calculations change based on number of dimensions, for example, in 3 dimensions it is sqrt(a^2+b^2+c^2).

Yes, that’s why it’s square root of 2. Consider the origin (0,0,0, …) to a random point on the sphere (~0, ~0, ~0, …). Distance = square root of ((X1 - X2) ^ 2 + (Y1 - Y2) ^2 + …). So D = square root of ((~0-0)^2 + (~0-0)^2 + (~0-0)^2 + … ), which is equal to 1 by definition of the unit high dimensional sphere. So distance from (1,0,0,0 …) to (~0, ~0, ~0, …) = square root of ((~0-1)^2 + (~0-0)^2 + (~0-0)^2 + … ) ~= s…

Ahh ok, for some reason I was thinking (1,1,1) would be a valid point in this case

Re: Computer scientists prove why bigger neural networks do better

#90

Post by visarga »

XnoiVeX wrote: post #76
cloogshicer wrote: post #37

Earlier quoted context omitted.

You probably already know this (since you wrote "silly thought"), but real-life neurons are ridiculously more complex than simulated "neurons" in an NN. So the analogy doesn't really hold.

I think a lot of people on this thread are missing this critical insight.

You can simulate the data processing of a real neuron with 1000 digital ones, a small neural net.

I think we read too much into the complexity of biological neurons. Remember they need to do much more than compute signals. They need to self assemble, self replicate and pass through various stages of growth. They need to function for 80-100 years. Many of those neurons and synapses exist only for redundancy and other biological constraints.

A digital neuron doesn't care about its physical substrate and can be millions of times faster. They can be copied identically for no cost and cheaply fine-tuned for new tasks. Their architecture and data can evolve much faster than ours, and the physical implementation can remain the same during this process.

Post reply on HN