Separating signal from noise in coding evaluations • Hacker News Board

Re: Separating signal from noise in coding evaluations

#61

Post by jbs789 » Thu, Jul 09, 2026, 12:28 AM UTC

donw wrote: post #47

kakugawa wrote: post #40

Earlier quoted context omitted.

The more subtle point is that there's a gap between the task and its verification. e.g. if you have an open-ended / under-specified prompt, the verification needs to be able to handle all potential solutions. So you can have a very narrow task prompt that's easy to verify (but likely too simple of a challenge). Or a more realistic task prompt that's much harder to verify. And likely harder to both build the robust ve…

A substantial portion of software engineering -- and the fundamental jobs of a proper Product Owner and UX Designer -- is to turn "vague ideas about what we need to do" into "this widget, on this page, it should work like this" It's not a pipeline, it's an ongoing conversation within any functional team, but this requires buy-in from management, who is often selected for "line must go up this quarter no matter the co…

Variance in time horizons explains a lot of corporate behaviour.

And it’s rational. We all have limited careers.

I think that all makes a bit more sense as we get older. Optimising for short time horizons is not what I strive for, but explains things.

Re: Separating signal from noise in coding evaluations

#62

Post by sk4rekr0w » Thu, Jul 09, 2026, 12:54 AM UTC

swyx wrote: post #24

dandaka wrote: post #10

What is considered SOTA for SWE benchmarks now?

https://cognition.ai/blog/frontier-code (disclaimer - was on the team - but also we covered swebench pro/deepswe issues in here as well.)

1.1 seems a lot better than the original release, which was a bit hyperbolic. excited to see the team keep iterating.

Re: Separating signal from noise in coding evaluations

#63

Post by [deleted] » Thu, Jul 09, 2026, 1:45 AM UTC

ACCount37 wrote: post #54

therobots927 wrote: post #51

Earlier quoted context omitted.

A literal bird brain would outperform an LLM on most spatial reasoning tasks. Extrapolating the core theory of LLMs - that we can reverse engineer reasoning through language - does that imply that if we train a bird song LLM to predict next “token” (pitch) of a birdsong, that the LLM could excel in a bird flight simulator? I think it’s pretty clear that this is a dead end.

Do birds expose enough of their cognition through birdsong? Do birds expose locomotion-relevant functions specifically through birdsong? Do we have enough birdsong data available to start solving the inverse problem? If "yes" on all, then we might be able to do it. I expect "no" on most of that, for birds. But humans treat language as an interface to their higher cognitive functions, and stockpile language data. That…

[deleted]

Re: Separating signal from noise in coding evaluations

#64

Post by therobots927 » Thu, Jul 09, 2026, 1:48 AM UTC

whackernews wrote: post #59

therobots927 wrote: post #41

Aren’t we past the point of needing benchmarks? If we’re as close to AGI as Sam says then the proof should be in the pudding. OpenAI should build a competing CRM / Figma / Photoshop with a couple dozen engineers and a Dyson sphere’s worth of compute and just prove the capabilities. This all feels like a 2024 re-run. Oh, ChatGPT is going to cure cancer? Then find ONE rare cancer and CURE IT. OpenAI has access to the b…

Yeh I totally agree. I’d go one step further and say even if one of these models cured cancer it’s still only going to have done it in a computer-y way doing computer stuff. Can that same model experience a beautiful landscape and convey its emotions in an evocative way, could it tell how you’re feeling when you come back from work after a hard day? Could it hop on one leg? What the hell even is AGI and how does it d…

If OpenAI cures ANY form of cancer then I will admit I was wrong.

Until then all we have is a lot of hot air coming out of Sam and Dario’s asses

Re: Separating signal from noise in coding evaluations

#65

Post by Der_Einzige » Thu, Jul 09, 2026, 1:49 AM UTC

schaefer wrote: post #56

SyneRyder wrote: post #49

Earlier quoted context omitted.

Seems like you're asking for the Artificial Analysis "Intelligence vs Cost" benchmark, perhaps? https://artificialanalysis.ai/?cost=intelligence-vs-cost-per...

It's still spooky to see exponential scales on the money axis. I do not have exponential funds in my allowance...

In terms of marginal utility you sure do.

Re: Separating signal from noise in coding evaluations

#66

Post by rao-v » Thu, Jul 09, 2026, 2:06 AM UTC

mlhpdx wrote: post #36

Fundamentally aren’t they concluding that tasks assigned to software developers (human or otherwise) are often incomplete, self contradictory or worse? This is the world in which their tool must play. I’m unsympathetic.

In a real job, you would be allowed to see the test case that failed and tweak your code (or more likely the poorly written test).

If you let a modern LLM do even the first, they’d crush this specific benchmark.

What is interesting is understanding how LLMs are able to beat 70+% on this benchmark or getting some of the poorly framed questions right? Are they implicitly learning the test writers style? Are the solutions leaking into their training set?

Perhaps reassuring is that even Fable stalls out at ~72% (on the hidden set which OpenAI did not run this analysis on), so perhaps training on the bench is not happening in anything but the most indirect ways.

I care a lot because small open models can never learn idiosyncrasies like this, so I really want good ways to judge models fairly.

EDIT: Humm OpenAI is muddying the water a bit. Only 20%ish of problems are broken in ways that are unfair to the agent, 4-10% are broken in favorable ways, so the benchmark ceiling is probably closer to 80-85%

Re: Separating signal from noise in coding evaluations

#67

Post by therobots927 » Thu, Jul 09, 2026, 2:19 AM UTC

ACCount37 wrote: post #54

therobots927 wrote: post #51

Earlier quoted context omitted.

A literal bird brain would outperform an LLM on most spatial reasoning tasks. Extrapolating the core theory of LLMs - that we can reverse engineer reasoning through language - does that imply that if we train a bird song LLM to predict next “token” (pitch) of a birdsong, that the LLM could excel in a bird flight simulator? I think it’s pretty clear that this is a dead end.

Do birds expose enough of their cognition through birdsong? Do birds expose locomotion-relevant functions specifically through birdsong? Do we have enough birdsong data available to start solving the inverse problem? If "yes" on all, then we might be able to do it. I expect "no" on most of that, for birds. But humans treat language as an interface to their higher cognitive functions, and stockpile language data. That…

I was meaning to imply that yes assuming we had a proportionate amount of birdsong data, would we be able to reverse engineer their flight abilities.

I think given the fact that spatial reasoning is nearly universal among species, we can very safely assume that it is “too evolutionary old and too low-level, too far removed from higher cognition, to leak into language heavily”

I think this is pretty apparent. It’s very rare for athletes to talk through their actions in high level detail - I saw the ball coming towards me at a 37 degree phi 23 degree epsilon angle at a speed of approximately 20 mph, I estimated it’s time to arrival would be .45 seconds etc. The eye-hand coordination occurs almost completely outside of what you consider conscious awareness. And it’s not easy to describe that’s why athletic coaching is difficult to do through words alone.

As far as ARC-AGI goes it looks like last years models were scoring https://arxiv.org/pdf/2505.11831

Frankly I don’t understand why you can’t train a multi-modal LLM on video game frame data. Is that just way too compute intensive to do? What am I missing here? Because I think it’s crazy to think that an LLM could learn to think spatially just from reading… even if they’re reading everything that’s ever been written. I think that about summarizes my position.

Re: Separating signal from noise in coding evaluations

#68

Post by therobots927 » Thu, Jul 09, 2026, 2:22 AM UTC

jpatten wrote: post #58

therobots927 wrote: post #51

Earlier quoted context omitted.

A literal bird brain would outperform an LLM on most spatial reasoning tasks. Extrapolating the core theory of LLMs - that we can reverse engineer reasoning through language - does that imply that if we train a bird song LLM to predict next “token” (pitch) of a birdsong, that the LLM could excel in a bird flight simulator? I think it’s pretty clear that this is a dead end.

Out of curiosity I gave Fable (on max effort) a CAD task yesterday, which was to design a space efficient carrying case for a set of fasteners in my repair kit for work. It used CadQuery to generate a STEP file. The result was pretty much exactly what I wanted, without needing any manual edits. I did go back and forth with it on the design, but was really impressed with the result. Without prompting it included nice…

Well the thing about CAD files is that through reinforcement learning you can basically ask the AI to generate the CAD file an arbitrary item - say it’s a rabbit. It might have examples of this already in its training set and it’s essentially a similarity lookup - but for sake of argument assume we are giving it examples at the edge of the distribution (the whole point of RL). It guesses and you render the file. You pass that image to another AI (not being trained) and ask it if it resembles the description you gave the AI in training. If it does, you have a positive example. If it doesn’t, negative. In that way you can essentially apply transfer learning from the image recognition functionality to the description -> CAD functionality.

But is that actually spatial reasoning? Or is it effectively image generation? Because there’s a difference. Spatial reasoning implies that you could drop it in a video game, give it rules, and let it run. And it would play the game well. Like a flight simulator. That would be true spatial reasoning because spatial reasoning is not just identifying objects but understanding how they interact with one another in a highly quantitative way.

Re: Separating signal from noise in coding evaluations

#69

Post by zvolsky » Thu, Jul 09, 2026, 2:39 AM UTC

The misleading prompt cases are inadvertently testing the model's ability to filter out noise from its instructions. That could be a benchmark on its own. The correct response is to flag the inconsistency and ask for clarification.

Re: Separating signal from noise in coding evaluations

#70

Post by ebcode » Thu, Jul 09, 2026, 2:52 AM UTC

jheitmann wrote: post #32

It reads to me like "We did all the work you'd do to figure out how to fix the benchmark, then we decided to throw out the benchmark". Is there some reason the underlying data is so golden that it can't be patched? At the end they argue for a slightly more curated approach to benchmark generation, but my gut is that using messy ill-specified tests taken from real world data and patching them into fairness would be a…

And it reads to me like they have some other reason to move on from SWE Bench Pro, but they don't want to say what it is. They say right up top, "~30% of the tasks are broken." But that leaves ~70% un-broken, which seems pretty good to me. It would be nice if they would also say: "Here's the list of instances that are broken: ". Or, "Here's the subset of SWE Bench Pro we will use going forward." They're letting the perfect be the enemy of the good.