Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Computer scientists prove why bigger neural networks do better

quantamagazine.org

Post reply on HN
Print view View original

131–140 of 151 posts

Re: Computer scientists prove why bigger neural networks do better

#131

Post by More-nitors »

gwern wrote: post #75
rackjack wrote: post #34

Silly thought: if bigger NN's are better, shouldn't more neurons be better? Why aren't elephants smarter than us, despite having more neurons? https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n... https://pubmed.ncbi.nlm.nih.gov/24971054/

Density is also important. If we look at other things - some recent studies have been done on number-counting ( https://royalsocietypublishing.org/doi/10.1098/rstb.2020.052... ) or bird brains ( https://www.gwern.net/docs/psychology/neuroscience/2020-herc... ) - density jumps out as a major predictor. African elephants may have some more neurons, but the density isn't as great as a human where it counts, so they are…

maybe this is like 20nm vs 10nm vs 5nm?

Re: Computer scientists prove why bigger neural networks do better

#132

Post by viraptor »

mensetmanusman wrote: post #129

Size = N x D D is the dimensionality, and they give a picture example with D = X.pixels * Y.pixels How does D change when you introduce video? (Picture arrays) E.g. can a NN recognize something more by having access to an array of approximately the same image as it sweeps through space/time?

You'd expect so, because there are already projects which do sub-pixel image recovery from video frame. So putting that before recognition should be doable.

Re: Computer scientists prove why bigger neural networks do better

#134

Post by akomtu »

WithinReason wrote: post #106
akomtu wrote: post #101

Earlier quoted context omitted.

"each individual xₙ will converge towards 0" I'm not sure it will. x1 is chosen randomly in the -1..1 interval. I dont see how the million other dimensions would force it to stick to 0. Those N other dimensions shrink the stddev(xi) by sqrt(N), though.

Then try normalizing a random 1000-element vector. The average of the vector elements is around 0.027.

The fact that average of D is sqrt(2) is the easy part. But average doesnt mean that random D values would concentrate in that spot. D may be evenly distributed. So the question is what's the variance of D? Using your formulas above:

D^2 = (1 - x1)^2 + (1 - x1^2) = 2 - 2x1

We know that x1 is 0 on average, but its distribution is restricted by the fact that we choose a random point on the n-sphere. So what's the probability of that random point falling into a thin stripe where abs(x1) This eps-stripe is basically the (n-1)-sphere of width eps, so the stripe's area A(n)=eps•S(n-1), and so our probability p=eps•S(n-1)/S(n).

The "magical" property of n-spheres is that their area grows at first, reaches maximum in 7 dimensions and then falls off rapidly to zero. Using formulas from wikipedia, I get: p = eps•(n/4)^(n/2) for large n.

In other words, the distribution of x1 approaches the look of the delta function at the n^n pace and for all practical matters, D=sqrt(2) with high precision for n > 10.

Where

Re: Computer scientists prove why bigger neural networks do better

#135

Post by t-3 »

tpoacher wrote: post #112
user90349032 wrote: post #79

Earlier quoted context omitted.

And yet, no animal except humans is self aware. Really makes you wonder why that is.

depends on your definition of self aware. Most cats and dogs I've seen correctly recognise themselves in the mirror after the novelty of seeing one for the first time wears off.

My cats use reflections to spy on and stalk each other, which seems to me a way more significant sign of intelligence than looking at themselves.

Re: Computer scientists prove why bigger neural networks do better

#136

Post by hikarudo »

SpaceManNabs wrote: post #3

> Right now, we are routinely creating neural networks that have a number of parameters more than the number of training samples. This says that the books have to be rewritten. Confused by this statement. Double descent with overparameterization is exhibited in "classical settings" too and mentioned in older books. > In their new proof, the pair show that overparameterization is necessary for a network to be robust.…

Here's a very good visual explanation of double descent:

https://mlu-explain.github.io/double-descent/

Re: Computer scientists prove why bigger neural networks do better

#138

Post by jfmc »

SpaceManNabs wrote: post #3

> Right now, we are routinely creating neural networks that have a number of parameters more than the number of training samples. This says that the books have to be rewritten. Confused by this statement. Double descent with overparameterization is exhibited in "classical settings" too and mentioned in older books. > In their new proof, the pair show that overparameterization is necessary for a network to be robust.…

> Right now, we are routinely creating neural networks that have a number of parameters more than the number of training samples. This says that the books have to be rewritten.

I also believe that this statement is weird. I have a very shallow knowledge of ML, but I can imagine that in a convolutional neural network a training sample interacts with lots of parameters. This 'one training sample one parameter' correspondence seems wrong to me.

Re: Computer scientists prove why bigger neural networks do better

#139

Post by urthor »

rackjack wrote: post #34

Silly thought: if bigger NN's are better, shouldn't more neurons be better? Why aren't elephants smarter than us, despite having more neurons? https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n... https://pubmed.ncbi.nlm.nih.gov/24971054/

If neural networks have shown us anything, it's that not all neurons are born the same. After all, even in nature there are multiple types of neurons.

I imagine it's 100% dependent on the cardinal rule of neural networks:

"Choice of training data is 10 times more important than the actual model."

What we have over elephants are opposable thumbs, excellent eyes, and vocal cords. And crucially, we're generally speaking pretty slow, weak and useless.

Except for our elaborate methods of I/O.

Our entire success is based on a feedback loop. "If human uses their IO this way, human will get more food."

Thus, we become ever more sophisticated at this. We are nothing if not a vehicle for using our high dexterity, low gross force, opposable thumbs in inventive ways to get food.

Plus, we have a biological imperative to pass these techniques on as knowledge.

A baby elephant can probably feed itself by eating green stuff at 1 year old (I know nothing about elephants).

A human child realistically cannot independently scrounge up enough solid food to sustain themselves, until they're what, twelve? Twenty-two? Certainly no younger than eight.

We have, almost certainly, the most useless progeny in the animal kingdom.

Hence we invest an enormous amount of time and energy in education to make them able to feed themselves.

So the tl;dr is

1) Human brains are pretty similar to the animal kingdom's. 2) Human opposable thumbs are world class. Pretty close to as good as it gets. Sight is also top notch, many animals have useless eyeballs. 3) Most human food is obtained by doing creative things with thumbs. This is very complex, and takes a lot of practice. 4) Human birth the most useless children in the entire animal kingdom. These children take decades to fully grow, hence we invest an enormous amount of time educating them in opposable thumbs. 5) Over time our education system gets better and better, and our list of clever things we can do with opposable thumbs get longer and longer.

Essentially what we have over the other animals isn't neurons.

What we have over the other animals is a data collection/cleaning/utilization cycle.

Re: Computer scientists prove why bigger neural networks do better

#140

Post by kaba0 »

988747 wrote: post #113
moomin wrote: post #82

Earlier quoted context omitted.

Probably that you don’t know how to measure what you’re describing. Plenty of animals recognise themselves in the mirror, for instance.

I wouldn't say "plenty" - few primates, dolphins, orcas, elephants, and, strangely, magpies. But the grounds for that claim are shaky for some of them, the only 3 species we are 100% sure about are chimpanzees, orangutans, and humans. Magpies, for example, require "a training" (whatever that means).

The mirror-test is biased to vision, though.

Dogs pass the scent-based mirror test for example, their eyes are just simply not the primary way of interacting with the world.

Post reply on HN