51–60 of 79 posts
The fact that this was on the set of training problems with a custom harness basically makes the headline a lie. What if you give opus the same harness? Do people even care about meaningful comparisons any more or is it all just “numbers go up”
Earlier quoted context omitted.
Benchmarks on public tests are too easy to game. The model owners can just incorporate the answers in to the dataset. Only the private problems actually matter.
In this case the code is public and you can see they are not cheating in that sense.
1. https://github.com/symbolica-ai/ARC-AGI-3-Agents/blob/symbol...
Earlier quoted context omitted.
We're calling agents harnesses now?
The point of this test is to check if an AI system can figure out the game. This isn't what happened here. A human figured out the game, wrote in their prompts exactly how the game works and THEN put the AI on the problem. This is 100% cheating and imo quite stupid.
Not sure if the specific rules of this prize allow that, but I would accept that
Earlier quoted context omitted.
It isn't arbitrary. They want measure the capability of the general LLM
So if I say "I want to measure your capability as a mechanic" but then also "to ensure an accurate score you're forbidden to use any tools" how are you the human mechanic planning to diagnose and fix the engine problem without wrenches and jack stands and the like? It makes no sense. That said their harness isn't generic. It includes a ridiculously detailed prompt for how to play this specific game. Forbidding tool u…
we constantly underestimate the power of inference scaffolding. I have seen it in all domains: coding, ASR, ARC-AGI benchmarks you name it. Scaffolding can do a lot! And post-training too. I am confident our currently pre-trained models can beat this benchmark over 80% with the right post-training and scaffolding. That being said I don't think ARC-AGI proves much. It is not a useful task at all in the wild. it is jus…
we constantly underestimate the power of inference scaffolding. I have seen it in all domains: coding, ASR, ARC-AGI benchmarks you name it. Scaffolding can do a lot! And post-training too. I am confident our currently pre-trained models can beat this benchmark over 80% with the right post-training and scaffolding. That being said I don't think ARC-AGI proves much. It is not a useful task at all in the wild. it is jus…
Earlier quoted context omitted.
what exactly does scaffolding mean in this context? genuine question
anything that doesn't touch the model parameters at all once it has been compiled. for example, in streaming ASR of an encoder-decoder you can get gains in accuracy just by enhancing the encoder-decoder orchestration and ratio, frequency of fwd passes, dynamically adjusting the length of rolling windows (if using full attention). Prompting would be part of this too, including few-shot examples. Decoding strategy is a…
Earlier quoted context omitted.
So if I say "I want to measure your capability as a mechanic" but then also "to ensure an accurate score you're forbidden to use any tools" how are you the human mechanic planning to diagnose and fix the engine problem without wrenches and jack stands and the like? It makes no sense. That said their harness isn't generic. It includes a ridiculously detailed prompt for how to play this specific game. Forbidding tool u…
It is more like restricting the mechanic to only using commercially available tools and not allow them to create CUSTOM tools.
The wrench is to the mechanic as the stock python repl is to the LLM.
Earlier quoted context omitted.
It is more like restricting the mechanic to only using commercially available tools and not allow them to create CUSTOM tools.
No, that would be analogous to disallowing customized harnesses, ie tooling specially crafted by someone else for the specific task at hand. Insisting that an LLM solve something without the ability to make use of any external tooling whatsoever is almost perfectly analogous to insisting that a human mechanic work on a car with nothing but his own bare hands. The wrench is to the mechanic as the stock python repl is…