Earlier quoted context omitted.
> I wonder how gpt-4 does. The problem is that as soon as people started tricking ChatGPT 3 into problems like that, the correct answers are now being used to train the next versions and are going to be part of the dataset. So GPT-4 or GPT-5 may get the answer right, but that still wouldn't mean anything.
Not the case for GPT-4 though, as it’s knowledge cutoff is the same as GPT-3s, that’s why it’s easy to compare the two on the same problems and see what’s the difference.
Yes, the GPT-4 paper says
> GPT-4 generally lacks knowledge of events that have occurred after the vast majority of its pre-training data cuts off in September 2021 [footnote: The pre-training and post-training data contain a small amount of more recent data], and does not learn from its experience.
But note the more recent data. We know that InstructGPT (GPT-3.5) was RL trained on examples of previous queries to GPT3, such as those trick questions. We could assume everything (after filtering e.g. for benchmark contamination) ever sent to OpenAI is in that post-training set. This is indeed a very small amount of data compared to the trillion-plus tokens of older data it was surely trained on. We also know that when ARC did their evaluations of GPT-4, OpenAI hadn't finished fine-tuning yet, so they've certainly been continuing to do so recently.
See also my other comment https://news.ycombinator.com/item?id=35300668