You can be reasonably confident that unless there’s been a significant breakthrough (there hasn’t) if a smaller model beats a larger model it’s either fine tuned for a specific purpose or trained on the test data somehow (ie. fine tuned to have good metrics).
To be less snarky, they claim:
> These models are built on a meticulously curated high-quality training corpus
Ie. Good training data plus a small model beats a bigger model.
…but I’m skeptical, when I read:
> We observed that performance continues to improve with longer training, even beyond the Chinchilla optimal point. Therefore, we decided to train the 1.7B model on 1 trillion tokens and the 135M and 360M models on 600B tokens, as the performance gains after 400B tokens begin to slow on some benchmarks for these smaller models.
So they’re evaluating their models against various benchmarks as they train them and picking the practice that gives the best benchmarks?
I dunno.
The claim is basically good data > more parameters, but it’s just an observation of “this happened to work for us” rather than something you can usefully take (as far as I can see) and apply to larger models.
The claims they actaully make about performance are far more modest than people are making out.
The 1.7B model performs better than any other 2B models in their evaluation.
Seems nice. Not ground breaking. Not convinced it’s real rather than polluted training data personally.