This doesn’t seem like opportune timing to announce days before a new model drop
Separating signal from noise in coding evaluations
21–30 of 108 posts
Re: Separating signal from noise in coding evaluations
#22Re: Separating signal from noise in coding evaluations
#23Earlier quoted context omitted.
SWE-Bench Pro was created to replace SWE-Bench and fix these problems.
SWE-bench Verified was created to fix the problems of SWE-bench. Then SWE-Bench Pro was created because SWE-bench Verified had flaws. Now SWE-Bench Pro is shown to have flaws.
Is there a way to benchmark the accuracy, validity improvements in these successive benchmarks?
Re: Separating signal from noise in coding evaluations
#24What is considered SOTA for SWE benchmarks now?
https://cognition.ai/blog/frontier-code (disclaimer - was on the team - but also we covered swebench pro/deepswe issues in here as well.)
Re: Separating signal from noise in coding evaluations
#25What is considered SOTA for SWE benchmarks now?
Either DeepSWE [0] or FrontierCode [1], depending on personal goals and requirements. The later is more interesting for me personally, due to the design of the benchmark heavily grading "mergability", i.e. how the provided output is to review and whether a serious developer can easily parse it and'd be willing to merge the result. In my mind and with my private evals, for quite some time I've held firm that a model can have a higher ceiling but that has limited value if I do not feel truly confident in signing off on the code.
Re: Separating signal from noise in coding evaluations
#26Re: Separating signal from noise in coding evaluations
#27Achieving AGI will be more than just passing all benchmarks, it has to account for the unknown problems too.
AGI is a long way off. Unless you’re talking about some unknown-to-me LLM marketing BS which is called “AGI” or something, I guess. Artificial general purpose intelligence is so different to LLMs or image AI that they are completely incomparable, except to say that they are all artificial. AGI will do a lot more than token prediction.
Re: Separating signal from noise in coding evaluations
#28Earlier quoted context omitted.
SWE-Bench Pro was created to replace SWE-Bench and fix these problems.
SWE-bench Verified was created to fix the problems of SWE-bench. Then SWE-Bench Pro was created because SWE-bench Verified had flaws. Now SWE-Bench Pro is shown to have flaws.
Well, we now have DeepSWE
Re: Separating signal from noise in coding evaluations
#29Re: Separating signal from noise in coding evaluations
#30Achieving AGI will be more than just passing all benchmarks, it has to account for the unknown problems too.
This ties into the bias-variance tradeoff ( https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff ) common with building non-LLM models. The solutions can only be a) figure out how to get LLMs smaller with similar performance so they don't memorize things/game the benchmarks and b) build benchmarks that are indeed comprehensive for all real-world data, which is infeasible.
I mean, people always say there are tradeoffs, until you reach the next frontier, in which there are tradeoffs at said frontier, and the next, and the next, etc.
In one sense, yes, tradeoffs are inescapable as the scope expands to the maximal possible scope. In another sense... it depends on the level of abstraction we're talking about.