141–150 of 151 posts
Silly thought: if bigger NN's are better, shouldn't more neurons be better? Why aren't elephants smarter than us, despite having more neurons? https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n... https://pubmed.ncbi.nlm.nih.gov/24971054/
> Right now, we are routinely creating neural networks that have a number of parameters more than the number of training samples. This says that the books have to be rewritten. Confused by this statement. Double descent with overparameterization is exhibited in "classical settings" too and mentioned in older books. > In their new proof, the pair show that overparameterization is necessary for a network to be robust.…
Here's a very good visual explanation of double descent: https://mlu-explain.github.io/double-descent/
Can you call this learning?
Silly thought: if bigger NN's are better, shouldn't more neurons be better? Why aren't elephants smarter than us, despite having more neurons? https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n... https://pubmed.ncbi.nlm.nih.gov/24971054/
Density is also important. If we look at other things - some recent studies have been done on number-counting ( https://royalsocietypublishing.org/doi/10.1098/rstb.2020.052... ) or bird brains ( https://www.gwern.net/docs/psychology/neuroscience/2020-herc... ) - density jumps out as a major predictor. African elephants may have some more neurons, but the density isn't as great as a human where it counts, so they are…
So there's a lower bound on the number of parameters required to produce a good interpolation of a broad class of "smooth" functions, and that's larger than the data size. Ok. I'm guessing one could find an even larger lower bound to approximate a more exotic class of functions that are still "smooth" in some intuitive sense, and it wouldn't make me any more excited.
The main problem is, what does it say about how well a neural network can approximate specific types of functions, compared to anything else, with the same or more parameters, with the same huge amount of data, which has always been the real mystery here?
Can we analytically define a class of functions that can be fit well by neural networks with n data points, using If not, then this paper doesn't explain anything better for neural networks than for say kernel methods with polynomial kernels or manifold reconstruction using restricted classes of spline functions. The sensationalist headline, along with the "books need to be written" rhetoric, wants to make us think the result or the techniques it presented could get us closer to answering the above questions, but it seems to me there's zero truth in that.
Earlier quoted context omitted.
And yet, no animal except humans is self aware. Really makes you wonder why that is.
that is simply incorrect bonobos, orcas, elephants, dolphins, chimpanzees, etc have all shown degrees of self awareness.
Earlier quoted context omitted.
And yet, no animal except humans is self aware. Really makes you wonder why that is.
Is that something that can/has been proven? From my understanding there are many other smart animals that are self aware.
We can already stop here and question, why does an upper bound on any similar probability interest us? If we want to show that the function $f$ exists with high probability, we should also consider lower bounds, not only the upper bounds as is done in the paper (clearly, I can bound any probability by 1 and would not be wrong).
But even leaving this question aside and going back to Theorem 2, they essentially show that for any sample of size $n$, they can find (and overfit) a smooth function $f$, given a sufficiently large model space $d$ and $k$. Assume I launch such a model in production, and continue generating further observations $n$. It follows from the theorem 2 that $f$ will quickly become unsuitable and require retraining on a larger training space $d$ and $k$.
However, from a statistical standing point, unless the data generating process is non-differentiable at every point, we should be able to assume that there exists N_epsilon, such, that for every n > n_epsilon, we should be able to find $f$ such that the errors would be controllably small (This is not at all what follows from the Theorem 2, suggesting that any such $f$ is still fitting on the errors, not necessarily the true process.
What am I missing (or assuming incorrectly)? Would be very interested to discuss this paper further!
Earlier quoted context omitted.
Seems like being a social animal is necessary for self awareness.
In fact there is a popular theory[1] that bird intelligence evolved because of the way their social structures work. Birds mate for life but they cheat . Every bird wants their partner to be loyal and itself to sex as many other birds as possible. This means birds have to keep track of who can and can't see them cheat, who knows and who doesn't. There's even evidence that they rat each other out (2nd degree info) if…
https://www.labroots.com/trending/plants-and-animals/15629/s...
> Right now, we are routinely creating neural networks that have a number of parameters more than the number of training samples. This says that the books have to be rewritten. Confused by this statement. Double descent with overparameterization is exhibited in "classical settings" too and mentioned in older books. > In their new proof, the pair show that overparameterization is necessary for a network to be robust.…
> Right now, we are routinely creating neural networks that have a number of parameters more than the number of training samples. This says that the books have to be rewritten. I also believe that this statement is weird. I have a very shallow knowledge of ML, but I can imagine that in a convolutional neural network a training sample interacts with lots of parameters. This 'one training sample one parameter' correspo…
Earlier quoted context omitted.
Density is also important. If we look at other things - some recent studies have been done on number-counting ( https://royalsocietypublishing.org/doi/10.1098/rstb.2020.052... ) or bird brains ( https://www.gwern.net/docs/psychology/neuroscience/2020-herc... ) - density jumps out as a major predictor. African elephants may have some more neurons, but the density isn't as great as a human where it counts, so they are…
My understanding is more than the density configuration of the neurons matters most. The reason is in some cases neural network with drop out's perform better than fully connected neural network. This proves less dense networks can be more intelligent.
Case in point: https://openai.com/blog/block-sparse-gpu-kernels/