Many of the datasets for the "benchmarks" on which the major public LLMs are assessed are clearly present in their training data, making them basically useless for establishing reliability of the models. Its fairly obvious that at least some of the improved scores from later generations of models are that this benchmark data is increasingly represented in the training data.
A better way of assessing LLMs is waiting a few weeks until novel tests have been created explicitly absent from all prior training data, and then using those.
As has been shown, eg., on legal test, exams, etc. performance drops off a cliff when future out-sample data is actually used. Rather than these faked pretend out-sample benchmarks.