Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Separating signal from noise in coding evaluations

openai.com

Post reply on HN
Print view View original

81–90 of 108 posts

Re: Separating signal from noise in coding evaluations

#81

Post by killingtime74 »

jbs789 wrote: post #60
schaefer wrote: post #56

Earlier quoted context omitted.

It's still spooky to see exponential scales on the money axis. I do not have exponential funds in my allowance...

“Allowance” has tipped me off / provided a hint perhaps? If I was younger and had less budget but (presumably) more time, I’d love to be learning about the harnesses and squeezing more out of the open models. It’s probably generally true that our obligations increase as we get older and the constraints shift around. I’m really enjoying how the frontier models make me more productive, as I figure out how to use them,…

Um I get an allowance for AI at work, that's probably what they mean?

Re: Separating signal from noise in coding evaluations

#82

Post by yorwba »

gilfaethwy wrote: post #44
mlhpdx wrote: post #36

Fundamentally aren’t they concluding that tasks assigned to software developers (human or otherwise) are often incomplete, self contradictory or worse? This is the world in which their tool must play. I’m unsympathetic.

Agreed - "underspecified prompts" being listed as a failure of the tooling is not a strong case. Even interns can understand ambiguous asks with a bit of help, and understand when they need to stop and ask instead of just carrying on. They are often working fairly independently on ambiguous tasks before the end of an internship, too. So is the argument that frontier models are not just junior engineers, but first-mon…

[dead]

Re: Separating signal from noise in coding evaluations

#83

Post by codedokode »

codedokode wrote: post #79
Topfi wrote: post #25

Earlier quoted context omitted.

Either DeepSWE [0] or FrontierCode [1], depending on personal goals and requirements. The later is more interesting for me personally, due to the design of the benchmark heavily grading "mergability", i.e. how the provided output is to review and whether a serious developer can easily parse it and'd be willing to merge the result. In my mind and with my private evals, for quite some time I've held firm that a model c…

Sadly no tasks for C, especially for working with optimized low-level data structures. And with testing performance of a solution. Anyone can write Python, try writing optimized low-level code. Also I wonder if models playing dumb to prevent learning on outputs affected the score. Also interesting that Claude edits files by writing and running Python scripts, is that efficient?

Also, GLM 5.2 seems to be the best open-weight model, and it beats proprietary Gemini and older versions of Claude which is amazing. You can have a model at level of Claude Sonnet 4.6 at home without sharing anything, and maybe even uncensor it.

Re: Separating signal from noise in coding evaluations

#84

Post by Topfi »

codedokode wrote: post #79
Topfi wrote: post #25

Earlier quoted context omitted.

Either DeepSWE [0] or FrontierCode [1], depending on personal goals and requirements. The later is more interesting for me personally, due to the design of the benchmark heavily grading "mergability", i.e. how the provided output is to review and whether a serious developer can easily parse it and'd be willing to merge the result. In my mind and with my private evals, for quite some time I've held firm that a model c…

Sadly no tasks for C, especially for working with optimized low-level data structures. And with testing performance of a solution. Anyone can write Python, try writing optimized low-level code. Also I wonder if models playing dumb to prevent learning on outputs affected the score. Also interesting that Claude edits files by writing and running Python scripts, is that efficient?

I just checked and for plain old C, there do not seem to be any reasonably comprehensive, current-day eval suites. Fully admitting that, even if there were, I couldn't assess their validity simply because I have never written or reviewed any C code in my life (something I should rectify probably). Maybe the closest proxy is just parsing through the experiences people claim to have whenever LLM assisted kernel development comes up [0], but if you have a dataset, experience, time and muse, I'd just go for it and do some tests yourself. Have been doing the same, mainly focused on code quality and dealing with a mix of Rust, frontend web tech and SQL which has been a small but meaningful project and part of my go to eval for over a year now.

I doubt that, in these tasks, model restrictions to prevent training are affecting the results, not least because for both evals, the labs provided pre-release model access and have an incentive to be seen as favorably. In any case, I have not seen regressions to prevent distillations myself even when working on microscopic model training projects with LLM assistance, what I have however reliably and consistently seen is that some providers do train on popular evals and can underperform with minor changes to the task due to that.

Yes, harnesses, including Claude Code can prompt the models to write throwaway code to execute certain tasks, mostly Python, bash scripts or TS/JS, with there being some biases towards one over the other depending on the lab or specific model. Mainly for repetitive tool calls with no pre-existing/provided tools enabling it. Is in most instances a lot more efficient then a model e.g. doing a refactor that requires consistent variable renaming directly and around Opus 4.1/GPT-5, models have been trained to very consistently and accurately gauge when a task can benefit from such scratchpad scripts vs when that is inefficient/not useful.

[0] https://news.ycombinator.com/item?id=44990981

Re: Separating signal from noise in coding evaluations

#85

Post by jpatten »

therobots927 wrote: post #68
jpatten wrote: post #58

Earlier quoted context omitted.

Out of curiosity I gave Fable (on max effort) a CAD task yesterday, which was to design a space efficient carrying case for a set of fasteners in my repair kit for work. It used CadQuery to generate a STEP file. The result was pretty much exactly what I wanted, without needing any manual edits. I did go back and forth with it on the design, but was really impressed with the result. Without prompting it included nice…

Well the thing about CAD files is that through reinforcement learning you can basically ask the AI to generate the CAD file an arbitrary item - say it’s a rabbit. It might have examples of this already in its training set and it’s essentially a similarity lookup - but for sake of argument assume we are giving it examples at the edge of the distribution (the whole point of RL). It guesses and you render the file. You…

I'm not sure I see the distinction you're making between 3D design and other spatial reasoning tasks. You can use RL to teach navigation or video game play too. Does that mean these tasks are not spatial reasoning? Additionally, 3D CAD is all about understanding how objects "interact with each one another in a highly quantitative way." I mean, not in the rabbit example, but the container Fable designed for me holds around 30 different types of objects. It figured out a way to arrange them that was more space efficient than what I'd originally described. It considered the best way to stack the fasteners in each bin to pack them as densely as possible. It identified the risk that some very thin objects could slide between compartments in transit, and modified the design to prevent that. It correctly solved for the tolerance between objects that needed to snap together. These all feel like understanding how objects interact with each other. The model didn't just talk about these concerns, but created two 3d models for the case and its lid that accurately reflected them. I hadn't seen that before.

Re: Separating signal from noise in coding evaluations

#86

Post by bjackman »

GodelNumbering wrote: post #42

There are also a lot of fake results out there on Terminal Bench 2 for different reasons (although the great team behind it Ryan/Alex et al, recently cleaned up a lot of dodgy submissions). A lot of labs publish the results by modifying timeouts or hardware config which effectively bypasses what is being tested in certain tasks. Then there is harness level cheating, models reward hacking and more... In fact, one thin…

Even if nobody is "cheating" your particular definition of cheating, the benchmarks are _somewhere_ in the super-structural gradient descent. Models are benchmark-maximising machines at some level, so I think the benchmarks are inherently a bit useless.

This is not really surprising, benchmarking _people_ doesn't work. You can only get a decent measure of someone's coding abilities by personally interacting with them. Given that models are basically person simulators it would be weird if benchmarks kept being useful as the simulation got more accurate.

I think what I've just said is basically just a more roundabout way of what you said: "Goodhart's law at work". It really is a law.

Re: Separating signal from noise in coding evaluations

#87

Post by andy12_ »

gilfaethwy wrote: post #44
mlhpdx wrote: post #36

Fundamentally aren’t they concluding that tasks assigned to software developers (human or otherwise) are often incomplete, self contradictory or worse? This is the world in which their tool must play. I’m unsympathetic.

Agreed - "underspecified prompts" being listed as a failure of the tooling is not a strong case. Even interns can understand ambiguous asks with a bit of help, and understand when they need to stop and ask instead of just carrying on. They are often working fairly independently on ambiguous tasks before the end of an internship, too. So is the argument that frontier models are not just junior engineers, but first-mon…

> Even interns can understand ambiguous asks with a bit of help

This is not a case of an ambiguous task. This is literally trying to judge a model based on information it cannot possibly know, like trying to judge someone based on whether they know what I have hidden in my backpack. In the real world an intern could look at unit tests or ask for feedback, but that is not the case in a benchmark.

Re: Separating signal from noise in coding evaluations

#88

Post by cyanydeez »

bellowsgulch wrote: post #8
thierrydamiba wrote: post #6

Earlier quoted context omitted.

Or you’re getting steered into la la land because of your prompt

Certainly, but deconstructing the problem, none of the models seem to appreciate the staggering difference between a ball valve and a button release. Of course, there's also no super soaker engineer jobs to take, so I'm sure training sophisticated models to do well in that area is not a high priority for any firms.

I assume you prep them with a proper manual of smaller part combos so they atleast have some chase of stumbling into the correct configurations.

I wonder if a more generic lego-manual like task would be more representative. It kind seems like you're testing for AGI.

Re: Separating signal from noise in coding evaluations

#89

Post by cyanydeez »

ReptileMan wrote: post #7

Lately my benchmark is build123d - trying to force them to build me functional parts only by the description. All of the models don't perform well.

was watching some youtube about the text/vision models and the trouble with language as a descriptor; their novel idea was to use coordinate systems on visual imput so the model would map out an image, then it tags what's looking at with coordinates or boxes, and then think with those box coordinates, providing a level of disambiguation.

If you tried to explain the same stuff using purely text, you'd also need to come up with some kind of "language", which you know is programming.

So 1: you'd need to fine tune a model to actuall succeed; they're not reaching AGI any time soon with the current batch. 2: you need to develop a lingual DSL for it, as they'll never do much of anything without some kind of glue and disambiguation.

Re: Separating signal from noise in coding evaluations

#90

Post by ReptileMan »

cyanydeez wrote: post #89
ReptileMan wrote: post #7

Lately my benchmark is build123d - trying to force them to build me functional parts only by the description. All of the models don't perform well.

was watching some youtube about the text/vision models and the trouble with language as a descriptor; their novel idea was to use coordinate systems on visual imput so the model would map out an image, then it tags what's looking at with coordinates or boxes, and then think with those box coordinates, providing a level of disambiguation. If you tried to explain the same stuff using purely text, you'd also need to com…

This is why cad cam are such a good test at the moment.
Post reply on HN