41–50 of 79 posts
The fact that this was on the set of training problems with a custom harness basically makes the headline a lie. What if you give opus the same harness? Do people even care about meaningful comparisons any more or is it all just “numbers go up”
we constantly underestimate the power of inference scaffolding. I have seen it in all domains: coding, ASR, ARC-AGI benchmarks you name it. Scaffolding can do a lot! And post-training too. I am confident our currently pre-trained models can beat this benchmark over 80% with the right post-training and scaffolding. That being said I don't think ARC-AGI proves much. It is not a useful task at all in the wild. it is jus…
what exactly does scaffolding mean in this context? genuine question
Knowing the nature of a test ahead of time, building out your capabilities and tooling before entering the exam hall when your peers don't have that advantage, makes you a cheater.
I wish we'd move past public test sets for LLM benchmarks: publish a plain english explanation of the tasks, allow questions and clarifications, and but never release a single question from the test set verbatim.
It made sense back when models needed to be finetuned on the task to even reliably answer. If we're saying this is the path to AGI we should be able to rely on the generalization of the model to get it right.
The fact that this was on the set of training problems with a custom harness basically makes the headline a lie. What if you give opus the same harness? Do people even care about meaningful comparisons any more or is it all just “numbers go up”
Knowing the nature of a test ahead of time, building out your capabilities and tooling before entering the exam hall when your peers don't have that advantage, makes you a cheater.
Lots of people doing the same with extra steps (generating synthetic data from test questions with the LLM then training on it) I wish we'd move past public test sets for LLM benchmarks: publish a plain english explanation of the tasks, allow questions and clarifications, and but never release a single question from the test set verbatim. It made sense back when models needed to be finetuned on the task to even relia…
This is the state of "AI" these days I guess...
[1] https://github.com/symbolica-ai/ARC-AGI-3-Agents/blob/symbol...
Earlier quoted context omitted.
Lots of people doing the same with extra steps (generating synthetic data from test questions with the LLM then training on it) I wish we'd move past public test sets for LLM benchmarks: publish a plain english explanation of the tasks, allow questions and clarifications, and but never release a single question from the test set verbatim. It made sense back when models needed to be finetuned on the task to even relia…
You have a problem with generating synthetic data from test questions? Humans simulate experiences in their mind. What's the problem?
If a model was trained on and you change the tokens passed to , it loses several 'IQ points' if it can even answer back at all anymore.
Synthetic data is fine. Synthetic data on very similar questions generated based on the description is typically fine. But once the shape of what you're training on gets too close to the actual holdout questions, you're getting an uplift that's not realistic for unseen tasks.
Uses public dataset to evaluate which is not meant for evaluation. Writes super specific prompt[1] and claims eye catching results. This is the state of "AI" these days I guess... [1] https://github.com/symbolica-ai/ARC-AGI-3-Agents/blob/symbol...
Earlier quoted context omitted.
> this uses a harness This seems like an arbitrary restriction. Tool-use requires a harness, and their whitepaper never defines exactly what counts as valid.
It isn't arbitrary. They want measure the capability of the general LLM
That said their harness isn't generic. It includes a ridiculously detailed prompt for how to play this specific game. Forbidding tool use is arbitrary and above all pointless hoop jumping but that doesn't make the linked "achievement" any less fraudulent.