Separating signal from noise in coding evaluations
1–10 of 108 posts
Re: Separating signal from noise in coding evaluations
#2Translation: other labs have learned to benchmaxx SWE-Bench Pro better than they do
Re: Separating signal from noise in coding evaluations
#3Achieving AGI will be more than just passing all benchmarks, it has to account for the unknown problems too.
Re: Separating signal from noise in coding evaluations
#4Seems like depending on your field these days, the hot thing to do is build your own private benchmarks.
In my own testing, no frontier model knows how to replicate an original 1990s Super Soaker prototype design, which for the most part, should be almost completely possible with Home Depot parts.
They just don't understand PVC parts, triggers, etc.
Re: Separating signal from noise in coding evaluations
#5Seems like depending on your field these days, the hot thing to do is build your own private benchmarks. In my own testing, no frontier model knows how to replicate an original 1990s Super Soaker prototype design, which for the most part, should be almost completely possible with Home Depot parts. They just don't understand PVC parts, triggers, etc.
Or defensively expect models to be stupid.
Seems the smart thing to do is not assume an agent will do the right thing. But to create the scaffold / harness that enforces constraints to steer them towards a good result.
Then you can swap out the really smart model for maybe something cheaper.
Re: Separating signal from noise in coding evaluations
#6Seems like depending on your field these days, the hot thing to do is build your own private benchmarks. In my own testing, no frontier model knows how to replicate an original 1990s Super Soaker prototype design, which for the most part, should be almost completely possible with Home Depot parts. They just don't understand PVC parts, triggers, etc.
Or you’re getting steered into la la land because of your prompt
Re: Separating signal from noise in coding evaluations
#7Lately my benchmark is build123d - trying to force them to build me functional parts only by the description. All of the models don't perform well.
Re: Separating signal from noise in coding evaluations
#8Seems like depending on your field these days, the hot thing to do is build your own private benchmarks. In my own testing, no frontier model knows how to replicate an original 1990s Super Soaker prototype design, which for the most part, should be almost completely possible with Home Depot parts. They just don't understand PVC parts, triggers, etc.
Or you’re getting steered into la la land because of your prompt
Certainly, but deconstructing the problem, none of the models seem to appreciate the staggering difference between a ball valve and a button release.
Of course, there's also no super soaker engineer jobs to take, so I'm sure training sophisticated models to do well in that area is not a high priority for any firms.
Re: Separating signal from noise in coding evaluations
#9Didn't we all know from the start that all of SWE-Bench was flawed? Even the authors concede the limitations and have long since moved on.
Re: Separating signal from noise in coding evaluations
#10What is considered SOTA for SWE benchmarks now?