Separating signal from noise in coding evaluations

Post by naikrovek » Fri, Jul 10, 2026, 2:01 AM UTC

ACCount37 wrote: post #43

naikrovek wrote: post #27

Earlier quoted context omitted.

AGI is a long way off. Unless you’re talking about some unknown-to-me LLM marketing BS which is called “AGI” or something, I guess. Artificial general purpose intelligence is so different to LLMs or image AI that they are completely incomparable, except to say that they are all artificial. AGI will do a lot more than token prediction.

What's your evidence of that? That AGI requires a truly novel architecture, and not just another iterative "LLM but with an extra trinket and wheels that spin ten times faster".

Post by retr0rocket » Fri, Jul 10, 2026, 7:18 PM UTC

esperent wrote: post #75

retr0rocket wrote: post #22

Earlier quoted context omitted.

Why is this a problem? Its like asking a person how many elder futhark runes are in the word strawberry. Unless you want to tack on bpe enconding table to every llm context its pointless

Coding contains many subtasks analogous to counting letters in a word accurately.

Post by aqfamnzc » Tue, Jul 14, 2026, 3:40 PM UTC

carabiner wrote: post #98

aqfamnzc wrote: post #97

Earlier quoted context omitted.

> Many benchmarks include the task of analyzing coding agent aptitude tests. This has led to bench-benchmarks comparing LLM test benchmarking methods, e.g. M Sampson (2025), PL Royle (2024). We performed a bench-bench-benchmark analysis using Mythos Ultra Max 9.6 of these bench-benchmarks. > Methods: We instructed various LLMs to perform the task "Write a prompt instructing a variety of LLMs to write a benchmark for…

Wait where is that from? Googling just takes me back here.

Post by kimjune01 » Tue, Jul 21, 2026, 7:03 PM UTC

Centigonal wrote: post #35

jheitmann wrote: post #32

It reads to me like "We did all the work you'd do to figure out how to fix the benchmark, then we decided to throw out the benchmark". Is there some reason the underlying data is so golden that it can't be patched? At the end they argue for a slightly more curated approach to benchmark generation, but my gut is that using messy ill-specified tests taken from real world data and patching them into fairness would be a…

If they fixed it, then it wouldn't be SWE-Bench Pro anymore, right? It'd be "SWE-Bench-Pro-Fixed-OpenAI." I think it's better optics for the independence of the benchmark if the OpenAI team lets some third party do the fixing and release the improved benchmark. ...Although OpenAI did exactly that when they released SWE-Bench Verified, so maybe I'm talking out of my butt here.