Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Separating signal from noise in coding evaluations

openai.com

Post reply on HN
Print view View original

51–60 of 108 posts

Re: Separating signal from noise in coding evaluations

#51

Post by therobots927 »

ACCount37 wrote: post #34
bellowsgulch wrote: post #4

Seems like depending on your field these days, the hot thing to do is build your own private benchmarks. In my own testing, no frontier model knows how to replicate an original 1990s Super Soaker prototype design, which for the most part, should be almost completely possible with Home Depot parts. They just don't understand PVC parts, triggers, etc.

It's a "commonsense spatial reasoning/problem solving" kind of problem. LLMs fail at spatial reasoning forever. What humans "easily" solve in seconds with raw spatial reasoning LLMs often find easier to solve by invoking A* or a constraint solver. Might be that text data is particularly bad at teaching that to LLMs. Or that being good at spatial reasoning requires true recurrence, and autoregressive chain of thought…

A literal bird brain would outperform an LLM on most spatial reasoning tasks.

Extrapolating the core theory of LLMs - that we can reverse engineer reasoning through language - does that imply that if we train a bird song LLM to predict next “token” (pitch) of a birdsong, that the LLM could excel in a bird flight simulator?

I think it’s pretty clear that this is a dead end.

Re: Separating signal from noise in coding evaluations

#52

Post by jumploops »

All of the benchmarks are pretty terrible when you look under the hood.

For context, I've been iterating on a "supervisor" to replace a lot of the rigamarole spent when working with Codex/Claude Code, and recently ran this agent against Terminal Bench 2.1

At first I was excited, because my spec-driven supervisor outperformed vanilla codex on a bunch of tasks, however as I looked deeper, I found a ton of issues with the tasks themselves.

The main takeaway is that the instructions are often ambiguous while the test cases are overly specific.

A few examples:

- For `configure-git-webserver` the task includes language like "so that I can run" which blurs the line between what the agent should deliver vs. what should be removed. This causes an overthinking agent to configure the server, and then remove the exact files that the verifier checks, because if the user were to run the same commands, they would conflict.

- For `make-mips-interpreter` the task includes the language "I will check that you booted doom correctly" which causes the agent to retain the generated file `/tmp/frame.bmp` because the supervisor expects the user to check that _it_ booted Doom correctly, not that Doom boots correctly in an isolated way. The verifier then fails to start Doom, because it exits when an existing `/tmp/frame.bmp` exists, not checking to see that it's created from the boot[0].

- For `mcmc-sampling-stan` the supervisor agent often reached the right value, but produced a domain-specific numeric output in scientific notation, rather than a simple decimal form. The verifier fails because it parses the result incorrectly[1].

These are just a few of the inconsistencies I've found, which leads me to believe that Terminal Bench 2.1 is already saturated, and the results from GPT-5.6 and Mythos are basically at the top of the expected threshold (88.8% and 88% respectively).

The biggest issue, as I can tell, is that most benchmarks are "one-shot" and rarely test the model+harness on long iteration tasks, which is the primary way most users use these tools in practice.

[0] https://github.com/harbor-framework/terminal-bench-2-1/issue...

[1] https://github.com/harbor-framework/terminal-bench-2-1/issue...

Re: Separating signal from noise in coding evaluations

#53

Post by jjcm »

SyneRyder wrote: post #49
jjcm wrote: post #46

I want a new bench - given $100 of api spend, how much can a model accomplish for a suite of benchmark tests? Give us something that measures a combination of efficiency and intelligence. I think this would allow for some interesting tactics for smaller models - eg they could do things like computer use to test their results and grind on problems for longer to verify the outputs, whereas larger models may not have bu…

Seems like you're asking for the Artificial Analysis "Intelligence vs Cost" benchmark, perhaps? https://artificialanalysis.ai/?cost=intelligence-vs-cost-per...

Not quite. These cost-per-task benchmarks report the cost of the task after the model gives its initial answer. The total cost is irrelevant, and isn't factored into the model's decisions - a run of the full benchmark for something like Fable might cost $10k.

What I'm looking for is the inverse. I want to give the model a budget of $100, and see how much it can accomplish with that $100. For smaller models, this means they can do more than just choose thinking amount, they can do something like a /loop to keep iterating on a problem until they get it right.

Can something like Deepseek V4 Flash get more answers correct than Fable, when given equal budgets?

Think of it as answering this question: How much intelligence can you get out of a model given a budget of $100? A cost-per-task dash correlates, but it doesn't give you an answer to that question.

Re: Separating signal from noise in coding evaluations

#54

Post by ACCount37 »

therobots927 wrote: post #51
ACCount37 wrote: post #34

Earlier quoted context omitted.

It's a "commonsense spatial reasoning/problem solving" kind of problem. LLMs fail at spatial reasoning forever. What humans "easily" solve in seconds with raw spatial reasoning LLMs often find easier to solve by invoking A* or a constraint solver. Might be that text data is particularly bad at teaching that to LLMs. Or that being good at spatial reasoning requires true recurrence, and autoregressive chain of thought…

A literal bird brain would outperform an LLM on most spatial reasoning tasks. Extrapolating the core theory of LLMs - that we can reverse engineer reasoning through language - does that imply that if we train a bird song LLM to predict next “token” (pitch) of a birdsong, that the LLM could excel in a bird flight simulator? I think it’s pretty clear that this is a dead end.

Do birds expose enough of their cognition through birdsong?

Do birds expose locomotion-relevant functions specifically through birdsong?

Do we have enough birdsong data available to start solving the inverse problem?

If "yes" on all, then we might be able to do it.

I expect "no" on most of that, for birds. But humans treat language as an interface to their higher cognitive functions, and stockpile language data. That looks an awful lot like a set of two "yes".

The last open question is: is there enough spatial reasoning reflected in the language data we have?

It's plausible that spatial reasoning is too evolutionary old and too low-level, too far removed from higher cognition, to leak into language heavily. And it's also plausible that existing LLM architecture is uniquely poorly suited to learning spatial reasoning - higher cognitive functions involved in things like writing code or even composing poetry are a better fit for the architecture. And it's plausible that we're underestimating just how complex spatial reasoning truly is - Moravec's paradox strikes again.

We know that LLMs perform poorly and improve slowly on spatial reasoning tasks, but not exactly why. And progress on things like ARC-AGI series shows that they're not completely inept.

Re: Separating signal from noise in coding evaluations

#55

Post by theLiminator »

naikrovek wrote: post #27
xacky wrote: post #3

Achieving AGI will be more than just passing all benchmarks, it has to account for the unknown problems too.

AGI is a long way off. Unless you’re talking about some unknown-to-me LLM marketing BS which is called “AGI” or something, I guess. Artificial general purpose intelligence is so different to LLMs or image AI that they are completely incomparable, except to say that they are all artificial. AGI will do a lot more than token prediction.

Please define AGI first.

Re: Separating signal from noise in coding evaluations

#56

Post by schaefer »

SyneRyder wrote: post #49
jjcm wrote: post #46

I want a new bench - given $100 of api spend, how much can a model accomplish for a suite of benchmark tests? Give us something that measures a combination of efficiency and intelligence. I think this would allow for some interesting tactics for smaller models - eg they could do things like computer use to test their results and grind on problems for longer to verify the outputs, whereas larger models may not have bu…

Seems like you're asking for the Artificial Analysis "Intelligence vs Cost" benchmark, perhaps? https://artificialanalysis.ai/?cost=intelligence-vs-cost-per...

It's still spooky to see exponential scales on the money axis.

I do not have exponential funds in my allowance...

Re: Separating signal from noise in coding evaluations

#58

Post by jpatten »

therobots927 wrote: post #51
ACCount37 wrote: post #34

Earlier quoted context omitted.

It's a "commonsense spatial reasoning/problem solving" kind of problem. LLMs fail at spatial reasoning forever. What humans "easily" solve in seconds with raw spatial reasoning LLMs often find easier to solve by invoking A* or a constraint solver. Might be that text data is particularly bad at teaching that to LLMs. Or that being good at spatial reasoning requires true recurrence, and autoregressive chain of thought…

A literal bird brain would outperform an LLM on most spatial reasoning tasks. Extrapolating the core theory of LLMs - that we can reverse engineer reasoning through language - does that imply that if we train a bird song LLM to predict next “token” (pitch) of a birdsong, that the LLM could excel in a bird flight simulator? I think it’s pretty clear that this is a dead end.

Out of curiosity I gave Fable (on max effort) a CAD task yesterday, which was to design a space efficient carrying case for a set of fasteners in my repair kit for work. It used CadQuery to generate a STEP file. The result was pretty much exactly what I wanted, without needing any manual edits. I did go back and forth with it on the design, but was really impressed with the result. Without prompting it included nice touches like ribs on the bottom of the lid to stop fasteners from migrating to adjacent compartments, and the right tolerance for the fit between the case and the lid. This is a dramatic improvement from Opus 4.8.

Re: Separating signal from noise in coding evaluations

#59

Post by whackernews »

therobots927 wrote: post #41

Aren’t we past the point of needing benchmarks? If we’re as close to AGI as Sam says then the proof should be in the pudding. OpenAI should build a competing CRM / Figma / Photoshop with a couple dozen engineers and a Dyson sphere’s worth of compute and just prove the capabilities. This all feels like a 2024 re-run. Oh, ChatGPT is going to cure cancer? Then find ONE rare cancer and CURE IT. OpenAI has access to the b…

Yeh I totally agree. I’d go one step further and say even if one of these models cured cancer it’s still only going to have done it in a computer-y way doing computer stuff. Can that same model experience a beautiful landscape and convey its emotions in an evocative way, could it tell how you’re feeling when you come back from work after a hard day? Could it hop on one leg? What the hell even is AGI and how does it differ from GI? I don’t know what we’re even talking about any more!

Re: Separating signal from noise in coding evaluations

#60

Post by jbs789 »

schaefer wrote: post #56
SyneRyder wrote: post #49

Earlier quoted context omitted.

Seems like you're asking for the Artificial Analysis "Intelligence vs Cost" benchmark, perhaps? https://artificialanalysis.ai/?cost=intelligence-vs-cost-per...

It's still spooky to see exponential scales on the money axis. I do not have exponential funds in my allowance...

“Allowance” has tipped me off / provided a hint perhaps?

If I was younger and had less budget but (presumably) more time, I’d love to be learning about the harnesses and squeezing more out of the open models.

It’s probably generally true that our obligations increase as we get older and the constraints shift around. I’m really enjoying how the frontier models make me more productive, as I figure out how to use them, so have more wiggle room on cost but less time.

Anyway… being nostalgic but I suspect I learned a tonne when cost was the constraint, but was less “productive”.

Post reply on HN