[flagged]
Separating signal from noise in coding evaluations
91–100 of 108 posts
Re: Separating signal from noise in coding evaluations
#92[flagged]
Re: Separating signal from noise in coding evaluations
#93[flagged]
Re: Separating signal from noise in coding evaluations
#94I can understand why this might make for a bad benchmark, but
Overly strict tests enforce specific implementation details not specified in the prompt, invalidating many functionally correct submissions.
Underspecified prompts omit requirements that hidden tests enforce and that are not reasonably inferable.
Low-coverage tests under check the requested feature, so incomplete fixes can pass.
A misleading prompt points models toward the wrong behavior or contradicts what tests require.
If the goal is, "how does my model compare to real SWEs", these are pretty reasonable situations that your model will have to encounter. It's a little like making a nursing exam and then flagging that some of the tests required you to ask the attending doctor for additional information that's not in the chart, or that the patient's family didn't fully explain their aging grandma's medical history.
I can understand why they might want a tighter benchmark, but if you're OpenAI and you promised your model as a replacement for real workers, this isn't the best look. It seems like you would want to test these things.
Re: Separating signal from noise in coding evaluations
#95Earlier quoted context omitted.
Certainly, but deconstructing the problem, none of the models seem to appreciate the staggering difference between a ball valve and a button release. Of course, there's also no super soaker engineer jobs to take, so I'm sure training sophisticated models to do well in that area is not a high priority for any firms.
I assume you prep them with a proper manual of smaller part combos so they atleast have some chase of stumbling into the correct configurations. I wonder if a more generic lego-manual like task would be more representative. It kind seems like you're testing for AGI.
Yeah, I start with smaller tasks, like matching dimensional parts. Tasks like these you can't one shot, or they end up producing diagrams of plausible looking Super Soakers, but maybe ones that look more like an enthusiast making their own designs from scratch. Fair approach! But replication is different.
For me, it's not knowing whether or not it understands there's a big difference between a ball valve and a button release, or that once you start talking about depressing mechanisms for pressure release, you're activating some sort of signals that are too close to triggers (which, is what you want, after all!) and "triggers" are embedded with a very short distance to "guns" in any well-trained model.
Re: Separating signal from noise in coding evaluations
#96Re: Separating signal from noise in coding evaluations
#97Earlier quoted context omitted.
SWE-bench Verified was created to fix the problems of SWE-bench. Then SWE-Bench Pro was created because SWE-bench Verified had flaws. Now SWE-Bench Pro is shown to have flaws.
Is there a way to benchmark the accuracy, validity improvements in these successive benchmarks?
> Many benchmarks include the task of analyzing coding agent aptitude tests. This has led to bench-benchmarks comparing LLM test benchmarking methods, e.g. M Sampson (2025), PL Royle (2024). We performed a bench-bench-benchmark analysis using Mythos Ultra Max 9.6 of these bench-benchmarks.
> Methods: We instructed various LLMs to perform the task "Write a prompt instructing a variety of LLMs to write a benchmark for benchmark analysis, including the instructions 'Create a benchmark for...
Re: Separating signal from noise in coding evaluations
#98Earlier quoted context omitted.
Is there a way to benchmark the accuracy, validity improvements in these successive benchmarks?
> Many benchmarks include the task of analyzing coding agent aptitude tests. This has led to bench-benchmarks comparing LLM test benchmarking methods, e.g. M Sampson (2025), PL Royle (2024). We performed a bench-bench-benchmark analysis using Mythos Ultra Max 9.6 of these bench-benchmarks. > Methods: We instructed various LLMs to perform the task "Write a prompt instructing a variety of LLMs to write a benchmark for…
Wait where is that from? Googling just takes me back here.
Re: Separating signal from noise in coding evaluations
#99Earlier quoted context omitted.
“Allowance” has tipped me off / provided a hint perhaps? If I was younger and had less budget but (presumably) more time, I’d love to be learning about the harnesses and squeezing more out of the open models. It’s probably generally true that our obligations increase as we get older and the constraints shift around. I’m really enjoying how the frontier models make me more productive, as I figure out how to use them,…
Um I get an allowance for AI at work, that's probably what they mean?
Really, that’s the term they use?
Man, I’d feel like a child if my boss gave me an “allowance”.
Wasn’t aware. Thank you.
Re: Separating signal from noise in coding evaluations
#100I want a new bench - given $100 of api spend, how much can a model accomplish for a suite of benchmark tests? Give us something that measures a combination of efficiency and intelligence. I think this would allow for some interesting tactics for smaller models - eg they could do things like computer use to test their results and grind on problems for longer to verify the outputs, whereas larger models may not have bu…
At whose prices?