Another recent (but not called out in this article) is the "Textbooks Are All You Need" paper [1]; the results seem to suggest that careful curation and curriculums of training data can significantly improve model capabilities (when training domain specific, smaller models). Claiming a 10x smaller model can outperform competitors. (Eg. phi-1 vs. starcoder) [1] https://arxiv.org/abs/2306.11644
TBH, it looks like metric manipulation to me. They have used GPT-3.5 to generate their data(and not use textbooks at all like the title suggests). And their dataset is very much like their benchmark data. While there was some filtering, but still it is very possible that lot of the benchmark questions were in training data. We likely wouldn't ever know how good the model is as it not only closed but they haven't prov…
From section 5:
In Figure 2.1, we see that training on CodeExercises leads to a substantial boost in the performance of the
model on the HumanEval benchmark. To investigate this boost, we propose to prune the CodeExercises
dataset by removing files that are “similar” to those in HumanEval. This process can be viewed as
a “strong form” of data decontamination. We then retrain our model on such pruned data, and still
observe strong performance on HumanEval. In particular, even after aggressively pruning more than
40% of the CodeExercises dataset (this even prunes files that are only vaguely similar to HumanEval, see
Appendix C), the retrained phi-1 still outperforms StarCoder.