[flagged]
tap tap tap tap tap
tap tap tap tap tap tap tap
Task-free intelligence testing of LLMs
11–20 of 25 posts
Re: Task-free intelligence testing of LLMs
#12Re: Task-free intelligence testing of LLMs
#13Game playing is the next frontier. Model economically valuable tasks as games and have the agents play/compete. Alphabench and Vendingbench show the potential of this approach.
A decade of reinforcement and agentic learning was spent playing games (Google Deepmind AlphaGo, AlphaStar, OpenAI Five), including against each other. So what makes it a new frontier?
Re: Task-free intelligence testing of LLMs
#14Typo:
"The behvior summary"
Re: Task-free intelligence testing of LLMs
#15Earlier quoted context omitted.
There's definitely more than "just" pattern matching in there - for example, current SOTA models 'plan ahead' to simultaneously process both rough outlines of an answer and specific subject details to then combine internally for the final result ( https://www.anthropic.com/research/tracing-thoughts-language... ).
Eh that is still encompassed by the term “pattern matching” in this context. Sure it’s complicated, but it’s still just a glorified spell checker.
I'm an LLM naysayer, and even I have no trouble seeing, or accepting, that they're much more than glorified spell checkers.
Re: Task-free intelligence testing of LLMs
#16On alternative ways to measure LLM intelligence, we had good success with this: https://arxiv.org/abs/2509.23510 In short: start with a dataset of question and answer pairs, where each question has been answered by two different LLMs. Ask the model you want to evaluate to choose the better answer for each pair. Then measure how consistently it selects winners. Does it reliably favor some models over the questions, or…
Doesn't that presume that one model dominates the other?
Re: Task-free intelligence testing of LLMs
#17Game playing is the next frontier. Model economically valuable tasks as games and have the agents play/compete. Alphabench and Vendingbench show the potential of this approach.
A decade of reinforcement and agentic learning was spent playing games (Google Deepmind AlphaGo, AlphaStar, OpenAI Five), including against each other. So what makes it a new frontier?
Its application to LLMs to push capabilities. We're going to tap out expert feedback, and objective/competitive arenas are going to be the only way to progress at a reasonable speed.
The difference is going to be instead of starting from pre-existing games and hoping that "generalizes" to intelligence, this time people are going to build gamified simulators of economically valuable stuff. This is feasible now because we can use LLMs to help generate these games much faster than we would have been able to previously.
Re: Task-free intelligence testing of LLMs
#18whats the assistant prompt being used for these? i dont think ive ever gotten these joking responses back to anything
Re: Task-free intelligence testing of LLMs
#19Earlier quoted context omitted.
There's definitely more than "just" pattern matching in there - for example, current SOTA models 'plan ahead' to simultaneously process both rough outlines of an answer and specific subject details to then combine internally for the final result ( https://www.anthropic.com/research/tracing-thoughts-language... ).
Eh that is still encompassed by the term “pattern matching” in this context. Sure it’s complicated, but it’s still just a glorified spell checker.
And we're just glorified oxidation. At some point the concept of "emergent systems" comes into play.