Task-free intelligence testing of LLMs • Hacker News Board

Re: Task-free intelligence testing of LLMs

#21

Post by rdos » Fri, Jan 09, 2026, 10:38 AM UTC

This is very interesting. Especially the last part where it shows gpt-5.2 and gpt-oss and their very similar and unique outcome of being 90%+ Serious.

I tested this locally and got the same result with gpt-oss 120b. But only on the default 'medium' reasoning effort. When I used 'low' I kept getting more playful responses with emojis and when I used 'high' I kept getting more guessing responses.

I had a lot of fun with this and it provided me with more insight than I would have thought.

Re: Task-free intelligence testing of LLMs

#22

Post by vintermann » Fri, Jan 09, 2026, 12:32 PM UTC

nestorD wrote: post #5

On alternative ways to measure LLM intelligence, we had good success with this: https://arxiv.org/abs/2509.23510 In short: start with a dataset of question and answer pairs, where each question has been answered by two different LLMs. Ask the model you want to evaluate to choose the better answer for each pair. Then measure how consistently it selects winners. Does it reliably favor some models over the questions, or…

Interesting, but couldn't a model "cheat" in this task by being very good at telling model outputs apart? How far do you get with a classifier simply trained to distinguish models by their output?

It seems to me many models - maybe by design - have a recognizable style which would be much easier to detect than evaluating the factual quality of answers.

Re: Task-free intelligence testing of LLMs

#23

Post by vintermann » Fri, Jan 09, 2026, 12:48 PM UTC

These aren't task free. They're just implicit task, "figure out what you're expected to do". These sort of riddle tasks are 100% dependent on who does the expecting.

This is not a new idea. Traditional IQ tests pivoted to them (they weren't originally like that), and no doubt they have great "discriminative power", because having the ability to figure out what's expected of you and not getting intimidated by cryptic and obtuse tasks put before you, are certainly extremely valuable skills in e.g business and politics.

But I always respected real tasks more. A question on a math test is honest; if it doesn't precisely define what's expected of you, the taskmaster has done a bad job, not you. It still can be extremely demanding.

An implicit task, by comparison, smells more of riddles, gnosticism. Do you know the way? Do you know the genre? (Once you know the genre of implicit tasks typical to IQ tests, you can easily increase your performance by a lot).

For that matter, this idea isn't new to machine learning either. Francois Chollet did it already, and he was IMO just as wrongheaded in thinking implicit tasks are somehow more indicative of "true intelligence" than explicit ones.

Re: Task-free intelligence testing of LLMs

#24

Post by nestorD » Fri, Jan 09, 2026, 6:55 PM UTC

esafak wrote: post #16

nestorD wrote: post #5

On alternative ways to measure LLM intelligence, we had good success with this: https://arxiv.org/abs/2509.23510 In short: start with a dataset of question and answer pairs, where each question has been answered by two different LLMs. Ask the model you want to evaluate to choose the better answer for each pair. Then measure how consistently it selects winners. Does it reliably favor some models over the questions, or…

Doesn't that presume that one model dominates the other?

It presumes some models are better than others (and we do find that providing data with a wide mix of model strengths improves convergence) but it does not need to be one model, and it does not even need to be transitive.

Re: Task-free intelligence testing of LLMs

#25

Post by nestorD » Fri, Jan 09, 2026, 6:58 PM UTC

vintermann wrote: post #22

nestorD wrote: post #5

On alternative ways to measure LLM intelligence, we had good success with this: https://arxiv.org/abs/2509.23510 In short: start with a dataset of question and answer pairs, where each question has been answered by two different LLMs. Ask the model you want to evaluate to choose the better answer for each pair. Then measure how consistently it selects winners. Does it reliably favor some models over the questions, or…

Interesting, but couldn't a model "cheat" in this task by being very good at telling model outputs apart? How far do you get with a classifier simply trained to distinguish models by their output? It seems to me many models - maybe by design - have a recognizable style which would be much easier to detect than evaluating the factual quality of answers.

In theory, yes! If this metric ever becomes a widely used standard, one would have to start accounting for that...

But, in practice, when asking a model to pick the best answer they see a single question / answers pair and focus on determining what they think is best.