Live data from Hacker News

Computer scientists prove why bigger neural networks do better

quantamagazine.org

101–110 of 151 posts

Re: Computer scientists prove why bigger neural networks do better

#101

Earlier quoted context omitted.

If you consider a point on the sphere it means choosing a bunch of xi such that: x1^2 + x2^2 + … + xn^2 = 1. Suppose wlog you pick (1,0,0,…,0). Then the distance from your point to a random point is: D = (x1-1)^2 + x2^2 + … + xn^2 And from the first equation we know: x1^2 = 1 - x2^2 - x3^2 - … - xn^2 Intuitionistically, your point will be far from a random point if x1 is close to zero, and x1 will be close to zero be…

Let me hijack your explanation starting from this point: D = (x₁-1)² + (x₂² + … + xₙ²) Since all the xₙ² sum to 1, as the dimensionality grows (∑xₙ²→1 as n→∞) each individual xₙ will converge towards 0. Since x₁ is almost 0, therefore the (x₁-1)² term will be almost 1. Since we know that ∑xₙ²=1, and that x₁² is almost 0, then we also know that ∑xₙ² - x₁² is almost 1, which is the 2nd half of the above expression for…

"each individual xₙ will converge towards 0"

I'm not sure it will. x1 is chosen randomly in the -1..1 interval. I dont see how the million other dimensions would force it to stick to 0. Those N other dimensions shrink the stddev(xi) by sqrt(N), though.

Re: Computer scientists prove why bigger neural networks do better

#102
post #38

Earlier quoted context omitted.

are we certain they're not? i'm not sure we know how to measure smartness

Dumbo is smarto! Elephants have bodies built like a tank (and used as such by Hannibal), but humans have better I/O ports. {reading, writing, listening, speaking, singing, typing, doing, going} Without opposable thumbs, an elephant is probably quite envious of human writing & typing. Let's use the privilege wisely to encourage one another, teach and learn from each other, from Donald Tusk, and give a helping hand.

But African elephants have quite versatile opposable finger-like extensions at the tip of their trunks (Asian elephants have only one such thing)

Re: Computer scientists prove why bigger neural networks do better

#103
post #88
post #84

Earlier quoted context omitted.

There are lots of self aware non human animals. Dolphins and elephants are famous examples, most primates as well. Even many birds show levels of self awareness and theory of mind (they know the difference between what they know and what others know)

Seems like being a social animal is necessary for self awareness.

You might be interested in the theories on the evolution of human intelligence: https://en.wikipedia.org/wiki/Evolution_of_human_intelligenc...

This is exactly the question the field is about, and I find it fascinating to read about

Re: Computer scientists prove why bigger neural networks do better

#104
post #84

Earlier quoted context omitted.

And yet, no animal except humans is self aware. Really makes you wonder why that is.

There are lots of self aware non human animals. Dolphins and elephants are famous examples, most primates as well. Even many birds show levels of self awareness and theory of mind (they know the difference between what they know and what others know)

There are also lots of non self-aware human animals :P

Re: Computer scientists prove why bigger neural networks do better

#105

Silly thought: if bigger NN's are better, shouldn't more neurons be better? Why aren't elephants smarter than us, despite having more neurons? https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n... https://pubmed.ncbi.nlm.nih.gov/24971054/

IANANS but my understanding is that neurons/body mass is more indicative. Large animals have more neurons because large bodies need more.

They talk about the encephalization quotient, which is to the 2/3 power

https://en.wikipedia.org/wiki/Encephalization_quotient

Re: Computer scientists prove why bigger neural networks do better

#106
post #101

Earlier quoted context omitted.

Let me hijack your explanation starting from this point: D = (x₁-1)² + (x₂² + … + xₙ²) Since all the xₙ² sum to 1, as the dimensionality grows (∑xₙ²→1 as n→∞) each individual xₙ will converge towards 0. Since x₁ is almost 0, therefore the (x₁-1)² term will be almost 1. Since we know that ∑xₙ²=1, and that x₁² is almost 0, then we also know that ∑xₙ² - x₁² is almost 1, which is the 2nd half of the above expression for…

"each individual xₙ will converge towards 0" I'm not sure it will. x1 is chosen randomly in the -1..1 interval. I dont see how the million other dimensions would force it to stick to 0. Those N other dimensions shrink the stddev(xi) by sqrt(N), though.

Then try normalizing a random 1000-element vector. The average of the vector elements is around 0.027.

Re: Computer scientists prove why bigger neural networks do better

#107

Silly thought: if bigger NN's are better, shouldn't more neurons be better? Why aren't elephants smarter than us, despite having more neurons? https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n... https://pubmed.ncbi.nlm.nih.gov/24971054/

For one thing, when the article says "bigger" it means "more parameters," not "more neurons."

Re: Computer scientists prove why bigger neural networks do better

#108
post #93
post #63

Earlier quoted context omitted.

That article doesn't seem to support your claim. All of the feats mentioned would be entirely unremarkable in your average human.

Really? You'd immediately recognize someone you knew for a few weeks over 20 years ago? You wouldn't need a bit to try to figure out who they are? If so, then your memory is unusually good. I know that this is well beyond my capabilities. Nor do I have the ability to visit a place that I lived 40 years earlier and find my way around.

> You'd immediately recognize someone you knew for a few weeks over 20 years ago?

Yeah? Maybe not if they were a kid 20 years ago or their appearance had otherwise changed significantly, but otherwise I don't see why not.

Re: Computer scientists prove why bigger neural networks do better

#109

Silly thought: if bigger NN's are better, shouldn't more neurons be better? Why aren't elephants smarter than us, despite having more neurons? https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n... https://pubmed.ncbi.nlm.nih.gov/24971054/

Elephants very likely could be more intelligent than us, it just seems that intelligence is a difficult thing to measure quantitatively.

It may be hard to measure and even define precisely, but I think it's pretty clear that if we did agree on a definition in the context of this conversation it would be defined in such a way that humans are more intelligent than elephants.

Re: Computer scientists prove why bigger neural networks do better

#110
So, the smoother the function, the easier it is to approximate using SGD? Does that mean there's a relationship between training time and parameters and beyond a certain parameter count we can discover an equivalent approximation by trading one for the other?
Post reply on HN