Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Computer scientists prove why bigger neural networks do better

quantamagazine.org

Post reply on HN
Print view View original

71–80 of 151 posts

Re: Computer scientists prove why bigger neural networks do better

#71

Post by lariati »

qorrect wrote: post #16
feketegy wrote: post #2

This looks interesting, I bookmarked it. My biggest blocker is the "statistics" part of M/L, knowing what algorithms to choose for various cases.

This book was a big help for me and is very well written, https://xcelab.net/rm/statistical-rethinking/ . You can find it free online ( along with video course ). The printed version is a very nice high quality book.

Thanks so much. That is an amazing level of choice in the example code. I need this right now as a type of statistical strength training.

Re: Computer scientists prove why bigger neural networks do better

#72

Post by Retric »

WithinReason wrote: post #43
dan-robertson wrote: post #30

Earlier quoted context omitted.

If you consider a point on the sphere it means choosing a bunch of xi such that: x1^2 + x2^2 + … + xn^2 = 1. Suppose wlog you pick (1,0,0,…,0). Then the distance from your point to a random point is: D = (x1-1)^2 + x2^2 + … + xn^2 And from the first equation we know: x1^2 = 1 - x2^2 - x3^2 - … - xn^2 Intuitionistically, your point will be far from a random point if x1 is close to zero, and x1 will be close to zero be…

Let me hijack your explanation starting from this point: D = (x₁-1)² + (x₂² + … + xₙ²) Since all the xₙ² sum to 1, as the dimensionality grows (∑xₙ²→1 as n→∞) each individual xₙ will converge towards 0. Since x₁ is almost 0, therefore the (x₁-1)² term will be almost 1. Since we know that ∑xₙ²=1, and that x₁² is almost 0, then we also know that ∑xₙ² - x₁² is almost 1, which is the 2nd half of the above expression for…

Close, the distance formula is square root of (X1^2 + X2^2 …).

So exactly 1 gives a distance of 1, but almost 1 + almost 1 gives a distance of almost square root of 2.

Re: Computer scientists prove why bigger neural networks do better

#73

Post by kd5bjo »

Is there a corresponding result that gives the number of examples needed to provide a sufficient training set for a given physical phenomenon? I’m imagining a high-dimensional equivalent of Nyquist’s sampling theorem.

Coupled with this result, we’d then have a reasonable estimator of the network size required for particular tasks before even starting the data collection.

Re: Computer scientists prove why bigger neural networks do better

#74

Post by pishpash »

tomrod wrote: post #23
aray wrote: post #14

Earlier quoted context omitted.

> Double descent with overparameterization is exhibited in "classical settings" too and mentioned in older books. I’m curious for references or citations to this. When I was going over double descent I tried to find citations like this (just in a couple places like ML/stats textbooks).

There are a handful of papers in the 90s that show this, but it wasn't recognized for what it is. Double descent is REALLY crazy to me, coming from a classical background.

Over-parameterization for regularization is really old. The pseudoinverse min-norm solution for under-determined linear systems even has that flavor.

Re: Computer scientists prove why bigger neural networks do better

#75

Post by gwern »

rackjack wrote: post #34

Silly thought: if bigger NN's are better, shouldn't more neurons be better? Why aren't elephants smarter than us, despite having more neurons? https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n... https://pubmed.ncbi.nlm.nih.gov/24971054/

Density is also important. If we look at other things - some recent studies have been done on number-counting (https://royalsocietypublishing.org/doi/10.1098/rstb.2020.052...) or bird brains (https://www.gwern.net/docs/psychology/neuroscience/2020-herc...) - density jumps out as a major predictor. African elephants may have some more neurons, but the density isn't as great as a human where it counts, so they are remarkably intelligent (like ravens and crows), but still not human-level. There are diminishing returns in both directions. We have more neurons than any bird as much or more dense, and we have more density than any elephant with as many or more neurons. Put that together, and we squeak across the finish line to being just smart enough to create civilization.

An analogy: what's the difference between a supercomputer, and the same number of CPUs scattered across a few datacenters? It's that in a supercomputer, those CPUs are packed physically as close as possible with expensive interconnects to allow them to communicate as fast as possible. (For many applications, the supercomputer will finish long before the spread out nodes ever finish communicating and idling.) But you need to improve both or else your new super-fast CPUs will spend all their time waiting on Infiniband to chug through, or your fancy new Infiniband will be underutilized and you should've bought more CPUs.

Re: Computer scientists prove why bigger neural networks do better

#76

Post by XnoiVeX »

cloogshicer wrote: post #37
rackjack wrote: post #34

Silly thought: if bigger NN's are better, shouldn't more neurons be better? Why aren't elephants smarter than us, despite having more neurons? https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n... https://pubmed.ncbi.nlm.nih.gov/24971054/

You probably already know this (since you wrote "silly thought"), but real-life neurons are ridiculously more complex than simulated "neurons" in an NN. So the analogy doesn't really hold.

I think a lot of people on this thread are missing this critical insight.

Re: Computer scientists prove why bigger neural networks do better

#77

Post by pishpash »

kd5bjo wrote: post #73

Is there a corresponding result that gives the number of examples needed to provide a sufficient training set for a given physical phenomenon? I’m imagining a high-dimensional equivalent of Nyquist’s sampling theorem. Coupled with this result, we’d then have a reasonable estimator of the network size required for particular tasks before even starting the data collection.

VC dimension?

Re: Computer scientists prove why bigger neural networks do better

#78

Post by WithinReason »

Retric wrote: post #72
WithinReason wrote: post #43

Earlier quoted context omitted.

Let me hijack your explanation starting from this point: D = (x₁-1)² + (x₂² + … + xₙ²) Since all the xₙ² sum to 1, as the dimensionality grows (∑xₙ²→1 as n→∞) each individual xₙ will converge towards 0. Since x₁ is almost 0, therefore the (x₁-1)² term will be almost 1. Since we know that ∑xₙ²=1, and that x₁² is almost 0, then we also know that ∑xₙ² - x₁² is almost 1, which is the 2nd half of the above expression for…

Close, the distance formula is square root of (X1^2 + X2^2 …). So exactly 1 gives a distance of 1, but almost 1 + almost 1 gives a distance of almost square root of 2.

Good point!

Re: Computer scientists prove why bigger neural networks do better

#79

Post by user90349032 »

gwern wrote: post #75
rackjack wrote: post #34

Silly thought: if bigger NN's are better, shouldn't more neurons be better? Why aren't elephants smarter than us, despite having more neurons? https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n... https://pubmed.ncbi.nlm.nih.gov/24971054/

Density is also important. If we look at other things - some recent studies have been done on number-counting ( https://royalsocietypublishing.org/doi/10.1098/rstb.2020.052... ) or bird brains ( https://www.gwern.net/docs/psychology/neuroscience/2020-herc... ) - density jumps out as a major predictor. African elephants may have some more neurons, but the density isn't as great as a human where it counts, so they are…

And yet, no animal except humans is self aware. Really makes you wonder why that is.
Post reply on HN