DeepMind’s New Language Model, Chinchilla

That seems pretty unlikely to me actually. As the models and training data get much bigger, I think the initial weights become less important (at least assuming your random weights have certain desirable statistical properties, which they do by construction usually).

Yeah, there are ~100B neurons, ~1Q synapses, but how much compute is the brain actually using over time?

Some quick googling gives this:

- Generation of an action potential seems to use ~2.5×10^−7 J [0]

- The brain consumes around 20W during normal activity

This seems to imply that there are around 8×10^7, call it 10^8, activations per second [1].

Apparently, the average neuron has 1000 synapses. Let's say each synapse requires 10 mulacc operations per activation. Doing that math gives about 10^12 FLOPs/s [2].

Integrate that over 18 years, and you get roughly 5.7×10^20 FLOPs [3].

PaLM required 2.56×10^24 FLOPs to train [4]. So, we have (way more than) enough compute, we're just not using it efficiently. We're wasting a lot of FLOPs on dense matrix multiplication.

There's plenty of wiggle room in these calculations. I checked over the math, but I'd appreciate if someone would let me know if I've missed something.

    [0]: https://link.springer.com/article/10.1007/s11571-018-9503-3
    [1]: https://www.wolframalpha.com/input?i2d=true&i=Divide%5B20+W%2C2.5%E2%80%89%C3%97%E2%80%89Power%5B10%2C%E2%88%927%5D+Joules%5D
    [2]: https://www.wolframalpha.com/input?i2d=true&i=Power%5B10%2C8%5D+Hz+*+1000+*+10+flop
    [3]: https://www.wolframalpha.com/input?i2d=true&i=Power%5B10%2C12%5D+Divide%5BFLOP%2Cs%5D+*+18+years
    [4]: https://blog.heim.xyz/palm-training-cost/#:~:text=PaLM%20(2022)-,2.5e24,-10x***

To elaborate on the sibling comment: main memory is much bigger, but CPUs are much, much slower. It would be a challenge to merely run a model like this on CPU, and totally infeasible to train one. So the challenge is to fit into the memory of a single GPU you can afford, coordinate multiple GPUs, or efficiently page from main memory into GPU.

The other thing with more parameters is that it gives the NN more ability to overfit. That means that instead of, say, learning what a dog is, it instead memorises all the sentences containing "dog" that it has ever seen.

(Side note: Transformers aren’t piecewise linear. The dot products are bilinear, and feeding in the same input twice (under different linear maps) into a bilinear map, produces a quadratic map, not a linear one.)

People arguing about this are basically all speaking ambiguously, in ways that tend to either make an apparent disagreement when there is none, or hide the location of the actual disagreement.

It is true that a piecewise-linear function, within any linear component, will have any convex combination of some points be sent to the corresponding convex combination of where it sends those points.

It is not true that a piecewise-linear model trained on a set of data points will produce only outputs which are a convex combination of outputs that appear in the training set.

These are both obvious.

If one person takes the former to be what “it just interpolates between points” means, and another takes the latter to be what “it doesn’t just interpolate between points” means, and then they argue about which of them is right, then both are being silly.

I’m not saying that this is literally what is happening. This is meant as a metaphor for somewhat more sophisticated/reasonable interpretations of “(doesn’t) just interpolate(s) between points in the data set”.

_____________

A model trained on images which produced only convex combinations of images in its training set, would clearly be producing what could be called “interpolations between images in its training set”, and taking convex combinations of images is unimpressive.

This is obviously not what today’s image ML models do.

And, of course, you aren’t claiming that they do.

______

I should speak plainly.

Much of where the disagreement is, or is hidden behind, is disagreement as to the meaning of “just interpolation”.

At one end, “just interpolation” could refer to “take the Voronoi cells of the inputs in the training set (or maybe the dual of it, whatever), and at runtime, find the nearest neighbors of the point and take the linear combination of their assigned outputs, weighted according to the distances to the point.” This would certainly be “interpolation”, and is not impressive, calling it “just interpolation” seems quite fitting. However, it is obviously not what ML models do.

On the other end of the scale, “interpolation” could be interpreted as meaning “any process whatsoever, except that the process is required to be based primarily on the training data, with the process being generated mechanically from the training data, of computing an output for a given input.” And, certainly today’s ML models satisfy this description, but with this description the moniker “just” seems, inappropriate. It is like saying “just a process”. Well, yeah, everything is a process.

__________

It seems to me like much of what the disagreement ought to be about (which might not be what it is about) is along the lines of, how many conceptual layers of something are captured? Like, say something modeled images of faces as “linear combinations of images from this list of images of faces”. That’s an extremely basic thing. Then, very slightly more, would be something that determines positions of facial features, and then does stretching etc. of images to make them line up with the image to reproduce, and then does linear combinations. Then, suppose something takes like, the parts of the images of the face which are just skin and not like lip skin or eyes, and takes local averages of this in a number of general locations of the face (relative to locations of facial features), and takes principal components of this across the training set (with principle components perhaps corresponding to perhaps, 1 or 2 for skin tone, and then directionality for the lighting in the image, and maybe a component for how shiny the skin is).

A model which represents a face in terms of variables which we can interpret as things like “position of eyes”, “skin tone”, “lighting”, seems notably less in the direction which one might call “interpolating” than one which just lists a coefficient for each image in the training set (or each principal component of images (taken as plain vectors) in the dataset). And, of course, one can go farther than this in this direction. And the further one goes in this direction, (so, like, the more that what the individual images in the training set tell the model is “here is more data about an overarching pattern”), the less it seems like what one might be inclined to call “just interpolation”.