Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Computer scientists prove why bigger neural networks do better

quantamagazine.org

Post reply on HN
Print view View original

91–100 of 151 posts

Re: Computer scientists prove why bigger neural networks do better

#91

Post by dr_dshiv »

user90349032 wrote: post #79
gwern wrote: post #75

Earlier quoted context omitted.

Density is also important. If we look at other things - some recent studies have been done on number-counting ( https://royalsocietypublishing.org/doi/10.1098/rstb.2020.052... ) or bird brains ( https://www.gwern.net/docs/psychology/neuroscience/2020-herc... ) - density jumps out as a major predictor. African elephants may have some more neurons, but the density isn't as great as a human where it counts, so they are…

And yet, no animal except humans is self aware. Really makes you wonder why that is.

Self awareness is social self awareness. Viewing oneself as a social actor.

Re: Computer scientists prove why bigger neural networks do better

#92

Post by Swizec »

visarga wrote: post #88
Swizec wrote: post #84

Earlier quoted context omitted.

There are lots of self aware non human animals. Dolphins and elephants are famous examples, most primates as well. Even many birds show levels of self awareness and theory of mind (they know the difference between what they know and what others know)

Seems like being a social animal is necessary for self awareness.

In fact there is a popular theory[1] that bird intelligence evolved because of the way their social structures work. Birds mate for life but they cheat. Every bird wants their partner to be loyal and itself to sex as many other birds as possible.

This means birds have to keep track of who can and can't see them cheat, who knows and who doesn't. There's even evidence that they rat each other out (2nd degree info) if they think there's a reward to be had. All of this requires immense intelligence, which happens to prove useful in other contexts.

There's also a bird species who does this with food caches. Easier to steal from others than to build their own so a plethora of deceptive tactics developed to ensure others can't see where you're storing those delicious nuts. Complete with fake caches, lying, and espionage.

[1] I learned about it in The Genius of Birds

Re: Computer scientists prove why bigger neural networks do better

#93

Post by btilly »

Ajedi32 wrote: post #63
btilly wrote: post #47

Earlier quoted context omitted.

How do you measure intelligence? Elephants have much better memories than we do! https://www.scientificamerican.com/article/elephants-never-f...

That article doesn't seem to support your claim. All of the feats mentioned would be entirely unremarkable in your average human.

Really? You'd immediately recognize someone you knew for a few weeks over 20 years ago? You wouldn't need a bit to try to figure out who they are?

If so, then your memory is unusually good. I know that this is well beyond my capabilities. Nor do I have the ability to visit a place that I lived 40 years earlier and find my way around.

Re: Computer scientists prove why bigger neural networks do better

#94

Post by dan-robertson »

Retric wrote: post #56
dan-robertson wrote: post #35

Earlier quoted context omitted.

The fact should say that the expected distance between two random points tends to the diameter as the dimension increases. The intuition is that to be close you need to be close in a large number of coordinates and the law of large numbers (though coordinates aren’t independent) suggests that is unlikely. If you fix one point on a sphere (say (1,0,…,0)) then, for a high dimension, most points will not have any extrem…

You forgot the square root on distance formula. Distance = square root of ((X1 - X2) ^ 2 + (Y1 - Y2) ^2 + …). Consider the origin (0,0,0, …) to a random point on the sphere (~0, ~0, ~0, …). So Distance from origin = square root of ((~0-0)^2 + (~0-0)^2 + (~0-0)^2 + … ), which sums to 1 by definition of the unit high dimensional sphere. Then plug in 1 vs 0 in the first place because we care about (1,0,0,0 …) and you ge…

Wow. Can’t believe I missed that.

Re: Computer scientists prove why bigger neural networks do better

#95

Post by Spooky23 »

btilly wrote: post #93
Ajedi32 wrote: post #63

Earlier quoted context omitted.

That article doesn't seem to support your claim. All of the feats mentioned would be entirely unremarkable in your average human.

Really? You'd immediately recognize someone you knew for a few weeks over 20 years ago? You wouldn't need a bit to try to figure out who they are? If so, then your memory is unusually good. I know that this is well beyond my capabilities. Nor do I have the ability to visit a place that I lived 40 years earlier and find my way around.

I think it depends on the intensity of the experience.

I recently found myself in a hotel that I stayed in as a 7-8 year old in the 80s for a particularly memorable vacation with my extended family. It was funny that I still remembered the I unusual aspects of the layout and could spot many of the changes that had been made over the years.

But if you asked me to describe someone I met for a few days in a business context in 2020, I’d have a hard time remembering detail.

Re: Computer scientists prove why bigger neural networks do better

#96

Post by tomrod »

pishpash wrote: post #74
tomrod wrote: post #23

Earlier quoted context omitted.

There are a handful of papers in the 90s that show this, but it wasn't recognized for what it is. Double descent is REALLY crazy to me, coming from a classical background.

Over-parameterization for regularization is really old. The pseudoinverse min-norm solution for under-determined linear systems even has that flavor.

Sure, but that's identification approaches in econometrics and matrix analysis contexts. Using that for neural networks is new-ish in the zeitgeist, which did not exist in the 1990s as it does today.

Re: Computer scientists prove why bigger neural networks do better

#97

Post by bee_rider »

notahacker wrote: post #53
bee_rider wrote: post #49

Earlier quoted context omitted.

In particular, a given elephant might be "more intelligent" than a human -- we just happen to have evolved from a particular niche that has rendered us bizarrely good at abstracting knowledge and combining it with the knowledge of other humans.

What is "more intelligent" if not "more capable of abstracting, synthesizing and sharing knowledge"?

How about the ability to solve novel problems?

We have very good problem solving ability of course, but a superpowered ability to ask others how they solved the problem. If we wanted to somehow define a kind of 'brain horsepower' type intelligence, it seems to me that the former is closer to it than the latter, and it doesn't seem obvious to me that humans would necessarily take the top spot. Or that there's a reasonable/ethical way to test it -- let's take a human, elephant, crow, and dolphin, raise them in total isolation from the any community to get a measure of their untrained intelligence... we might get some interesting results on intelligence, but mostly we will learn something about ballistics as some ethics review board launches us unto the Sun.

Re: Computer scientists prove why bigger neural networks do better

#98

Post by andrewflnr »

fabiospampinato wrote: post #70
rackjack wrote: post #34

Silly thought: if bigger NN's are better, shouldn't more neurons be better? Why aren't elephants smarter than us, despite having more neurons? https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n... https://pubmed.ncbi.nlm.nih.gov/24971054/

I read on wikipedia [0] the other day a fairly disturbing statistic related to this, apparently human men have on average a ~10% bigger brain than women. It'd be interesting to know if that translates to a higher neuron count or the difference in volume is due to something else. [0]: https://en.wikipedia.org/wiki/Brain_size#:~:text=In%20men%20... .

Probably just a consequence of overall physical size being larger. AFAIK there continues to be no evidence of a sex difference in overall intelligence, so slight difference in brain size is probably a red herring.

Re: Computer scientists prove why bigger neural networks do better

#99

Post by throwmeawaysoon »

SpaceManNabs wrote: post #3

> Right now, we are routinely creating neural networks that have a number of parameters more than the number of training samples. This says that the books have to be rewritten. Confused by this statement. Double descent with overparameterization is exhibited in "classical settings" too and mentioned in older books. > In their new proof, the pair show that overparameterization is necessary for a network to be robust.…

No post body was provided.

Re: Computer scientists prove why bigger neural networks do better

#100

Post by Someone »

btilly wrote: post #93
Ajedi32 wrote: post #63

Earlier quoted context omitted.

That article doesn't seem to support your claim. All of the feats mentioned would be entirely unremarkable in your average human.

Really? You'd immediately recognize someone you knew for a few weeks over 20 years ago? You wouldn't need a bit to try to figure out who they are? If so, then your memory is unusually good. I know that this is well beyond my capabilities. Nor do I have the ability to visit a place that I lived 40 years earlier and find my way around.

How many other elephants did these elephants see in those 20+ years? It wouldn’t surprise me if that was fewer than 100. How many did they spend a few weeks or more with? It wouldn’t surprise me if that were less than 20.

There also, AFAIK, isn’t evidence they remember _all_ other elephants they’ve shared time with for at least few weeks (I certainly do not rule that out, either, given the low number they likely will meet in their life)

Post reply on HN