Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Separating signal from noise in coding evaluations

openai.com

Post reply on HN
Print view View original

71–80 of 108 posts

Re: Separating signal from noise in coding evaluations

#71

Post by qznc »

jjcm wrote: post #53
SyneRyder wrote: post #49

Earlier quoted context omitted.

Seems like you're asking for the Artificial Analysis "Intelligence vs Cost" benchmark, perhaps? https://artificialanalysis.ai/?cost=intelligence-vs-cost-per...

Not quite. These cost-per-task benchmarks report the cost of the task after the model gives its initial answer. The total cost is irrelevant, and isn't factored into the model's decisions - a run of the full benchmark for something like Fable might cost $10k. What I'm looking for is the inverse. I want to give the model a budget of $100, and see how much it can accomplish with that $100. For smaller models, this mean…

Like this, i guess: https://entropicthoughts.com/updated-llm-benchmark

Just for coding instead of text adventures.

Re: Separating signal from noise in coding evaluations

#72

Post by mkozlows »

mlhpdx wrote: post #36

Fundamentally aren’t they concluding that tasks assigned to software developers (human or otherwise) are often incomplete, self contradictory or worse? This is the world in which their tool must play. I’m unsympathetic.

This was also my thought, and I think holds true of the ones with invisible requirements that aren't stated up front and are only captured in tests. Oh, you need to rework your solution to handle requirements nobody mentioned before? Well, me too.

Yeah, it's testing a different thing than what the benchmark claims to test, but it's also accidentally testing something more real-world applicable than a clean benchmark would be, so hey.

(EDIT: That is, if the agent is allowed to see the failed tests and iterate. If not, then yeah, that's just a problem. And either way, the ones with tests that just encode a particular solution's implementation details, thereby demanding that your solution have some rando internal details, are junkier. That's not a situation you'd run into in reality.)

Re: Separating signal from noise in coding evaluations

#74

Post by esperent »

ebcode wrote: post #70
jheitmann wrote: post #32

It reads to me like "We did all the work you'd do to figure out how to fix the benchmark, then we decided to throw out the benchmark". Is there some reason the underlying data is so golden that it can't be patched? At the end they argue for a slightly more curated approach to benchmark generation, but my gut is that using messy ill-specified tests taken from real world data and patching them into fairness would be a…

And it reads to me like they have some other reason to move on from SWE Bench Pro, but they don't want to say what it is. They say right up top, "~30% of the tasks are broken." But that leaves ~70% un-broken, which seems pretty good to me. It would be nice if they would also say: "Here's the list of instances that are broken: ". Or, "Here's the subset of SWE Bench Pro we will use going forward." They're letting the p…

I think you can be sure they would have done that if it showed their model on or very close to the top.

Re: Separating signal from noise in coding evaluations

#75

Post by esperent »

retr0rocket wrote: post #22
carabiner wrote: post #15

Earlier quoted context omitted.

strawberry

Why is this a problem? Its like asking a person how many elder futhark runes are in the word strawberry. Unless you want to tack on bpe enconding table to every llm context its pointless

Coding contains many subtasks analogous to counting letters in a word accurately.

Re: Separating signal from noise in coding evaluations

#76

Post by SyneRyder »

therobots927 wrote: post #64
whackernews wrote: post #59

Earlier quoted context omitted.

Yeh I totally agree. I’d go one step further and say even if one of these models cured cancer it’s still only going to have done it in a computer-y way doing computer stuff. Can that same model experience a beautiful landscape and convey its emotions in an evocative way, could it tell how you’re feeling when you come back from work after a hard day? Could it hop on one leg? What the hell even is AGI and how does it d…

If OpenAI cures ANY form of cancer then I will admit I was wrong. Until then all we have is a lot of hot air coming out of Sam and Dario’s asses

Are you familiar with the Rosie the dog story?

AI designs cancer vaccine for dog but scientists says red tape a barrier for human care

https://www.abc.net.au/news/2026-06-22/australian-dog-cancer...

People have already been using ChatGPT to design custom bespoke mRNA vaccines specifically for one patient, based on sequencing their specific cancer. It already works to reduce tumours. Sam and Dario know this, it's why they can make their claims - it's already done. The problem is the cost of the procedure (which is why only rich entrepreneurs are seeing their cancers treated this way so far) and government regulations preventing its use in wider human populations without a 10 year study first.

Re: Separating signal from noise in coding evaluations

#77

Post by ACCount37 »

therobots927 wrote: post #67
ACCount37 wrote: post #54

Earlier quoted context omitted.

Do birds expose enough of their cognition through birdsong? Do birds expose locomotion-relevant functions specifically through birdsong? Do we have enough birdsong data available to start solving the inverse problem? If "yes" on all, then we might be able to do it. I expect "no" on most of that, for birds. But humans treat language as an interface to their higher cognitive functions, and stockpile language data. That…

I was meaning to imply that yes assuming we had a proportionate amount of birdsong data, would we be able to reverse engineer their flight abilities. I think given the fact that spatial reasoning is nearly universal among species, we can very safely assume that it is “too evolutionary old and too low-level, too far removed from higher cognition, to leak into language heavily” I think this is pretty apparent. It’s ver…

And today's records on ARC-AGI-2 are >80%. Held by LLMs that use text modality for input.

The issue with multimodal training is that it doesn't seem to bring a step-change improvement in spatial reasoning either. It helps some, but the gain is surprisingly small compared to the data and compute expended. What it helps with the most is, unsurprisingly, spatial reasoning when using image inputs.

Maybe there are gains we don't know how to extract there.

Overall, LLM performance at spatial tasks is improving, especially on things like puzzles, but that mix of "commonsense + spatial" in the same task still eludes them.

Re: Separating signal from noise in coding evaluations

#78

Post by elictronic »

jaggederest wrote: post #39
carabiner wrote: post #23

Earlier quoted context omitted.

Is there a way to benchmark the accuracy, validity improvements in these successive benchmarks?

Bench Bench Pro Maxx Series S 360? The original Bench Bench Pro Maxx Series S had some quality issues, so that's the current followup. We've also released a higher order benchmark developed out of Bench Bench Pro Maxx Series S 360 One King Ranch edition, allowing future benchmark towers to be fully self-contained.

Boo, I thought you were going for Street fighter references at first.

Re: Separating signal from noise in coding evaluations

#79

Post by codedokode »

Topfi wrote: post #25
dandaka wrote: post #10

What is considered SOTA for SWE benchmarks now?

Either DeepSWE [0] or FrontierCode [1], depending on personal goals and requirements. The later is more interesting for me personally, due to the design of the benchmark heavily grading "mergability", i.e. how the provided output is to review and whether a serious developer can easily parse it and'd be willing to merge the result. In my mind and with my private evals, for quite some time I've held firm that a model c…

Sadly no tasks for C, especially for working with optimized low-level data structures. And with testing performance of a solution. Anyone can write Python, try writing optimized low-level code.

Also I wonder if models playing dumb to prevent learning on outputs affected the score.

Also interesting that Claude edits files by writing and running Python scripts, is that efficient?

Post reply on HN