What is considered SOTA for SWE benchmarks now?
I've generally found DeepSWE[0] to be pretty true to reality.
Separating signal from noise in coding evaluations
11–20 of 108 posts
Re: Separating signal from noise in coding evaluations
#12Lately my benchmark is build123d - trying to force them to build me functional parts only by the description. All of the models don't perform well.
IDK, sounds like it has brute forced my password already.
Re: Separating signal from noise in coding evaluations
#13What is considered SOTA for SWE benchmarks now?
FrontierBench
Re: Separating signal from noise in coding evaluations
#14Achieving AGI will be more than just passing all benchmarks, it has to account for the unknown problems too.
Unless they have something in the labs that massively departs from their current products, AGI isn't on the table and is purely hype for marketing purposes.
Re: Separating signal from noise in coding evaluations
#15What is considered SOTA for SWE benchmarks now?
strawberry
Re: Separating signal from noise in coding evaluations
#16Didn't we all know from the start that all of SWE-Bench was flawed? Even the authors concede the limitations and have long since moved on.
SWE-Bench Pro was created to replace SWE-Bench and fix these problems.
Re: Separating signal from noise in coding evaluations
#17Interesting timing to release this just when SWE-1.7 and Grok 4.5 came out being much cheaper than GPT-5.5.
Re: Separating signal from noise in coding evaluations
#18Didn't we all know from the start that all of SWE-Bench was flawed? Even the authors concede the limitations and have long since moved on.
SWE-Bench Pro was created to replace SWE-Bench and fix these problems.
SWE-bench Verified was created to fix the problems of SWE-bench.
Then SWE-Bench Pro was created because SWE-bench Verified had flaws.
Now SWE-Bench Pro is shown to have flaws.
Re: Separating signal from noise in coding evaluations
#19Achieving AGI will be more than just passing all benchmarks, it has to account for the unknown problems too.
they should be consulting Donald Rumsfeld and make sure they implement the Unknown-Unknowns benchmark, because thats how they get you
Re: Separating signal from noise in coding evaluations
#20Achieving AGI will be more than just passing all benchmarks, it has to account for the unknown problems too.
This ties into the bias-variance tradeoff (https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff) common with building non-LLM models. The solutions can only be a) figure out how to get LLMs smaller with similar performance so they don't memorize things/game the benchmarks and b) build benchmarks that are indeed comprehensive for all real-world data, which is infeasible.