Qodo CLI agent scores 71.2% on SWE-bench Verified
1–10 of 60 posts
Re: Qodo CLI agent scores 71.2% on SWE-bench Verified
#2I’ve been running a bunch of coding agents on benchmarks recently as part of consulting, and this is actually much more impressive than it seems at first glance.
71.2% puts it at 5th, which is 4 points below the leader (four points is a lot) and just over 1% lower than Anthropic’s own submission for Claude Sonnet 4 - the same model these guys are running.
But the top rated submissions aren’t running production products. They generally have extensive scaffolding or harnesses that were built *specifically for SWE bench*, which kind of defeats the whole purpose of the benchmark.
Take for example Refact which is at #2 with 74.4%, they built a 2k lines of code framework around their agent specifically for SWE bench (https://github.com/smallcloudai/refact-bench/). It’s pretty elaborate, orchestrating multiple agents, with a debug agent that kicks in if the main agent fails. The debug agent analyzes the failure and gives insights to the main agent which tries again, so it’s effectively multiple attempts per problem.
If the results can be reproduced “out-of-the-box” with their coding agent like they claim, it puts it up there as one of the top 2-3 CLI agents available right now.
Re: Qodo CLI agent scores 71.2% on SWE-bench Verified
#3I’ve been running a bunch of coding agents on benchmarks recently as part of consulting, and this is actually much more impressive than it seems at first glance. 71.2% puts it at 5th, which is 4 points below the leader (four points is a lot) and just over 1% lower than Anthropic’s own submission for Claude Sonnet 4 - the same model these guys are running. But the top rated submissions aren’t running production produc…
According to your experience with this model, is it just trained for the benchmark or these points are actually representing the performance?
Re: Qodo CLI agent scores 71.2% on SWE-bench Verified
#4I’ve been running a bunch of coding agents on benchmarks recently as part of consulting, and this is actually much more impressive than it seems at first glance. 71.2% puts it at 5th, which is 4 points below the leader (four points is a lot) and just over 1% lower than Anthropic’s own submission for Claude Sonnet 4 - the same model these guys are running. But the top rated submissions aren’t running production produc…
What are the typical context lengths in SWE-bench problems? Does it partly measure performance in the 64-128k context range?
Re: Qodo CLI agent scores 71.2% on SWE-bench Verified
#5I’ve been running a bunch of coding agents on benchmarks recently as part of consulting, and this is actually much more impressive than it seems at first glance. 71.2% puts it at 5th, which is 4 points below the leader (four points is a lot) and just over 1% lower than Anthropic’s own submission for Claude Sonnet 4 - the same model these guys are running. But the top rated submissions aren’t running production produc…
According to your experience with this model, is it just trained for the benchmark or these points are actually representing the performance?
[deleted]
Re: Qodo CLI agent scores 71.2% on SWE-bench Verified
#6[dead]
Re: Qodo CLI agent scores 71.2% on SWE-bench Verified
#7I’ve been running a bunch of coding agents on benchmarks recently as part of consulting, and this is actually much more impressive than it seems at first glance. 71.2% puts it at 5th, which is 4 points below the leader (four points is a lot) and just over 1% lower than Anthropic’s own submission for Claude Sonnet 4 - the same model these guys are running. But the top rated submissions aren’t running production produc…
What are the typical context lengths in SWE-bench problems? Does it partly measure performance in the 64-128k context range?
IIRC the SWE bench dataset gives you the full repo snapshot + the issue text, the evaluation pipelines typically run some kind of retriever (eg. grep, BM25) to pick a subset of files to place in the model’s context. They provided context is usually limited up to ~50k tokens.
Re: Qodo CLI agent scores 71.2% on SWE-bench Verified
#8I’ve been running a bunch of coding agents on benchmarks recently as part of consulting, and this is actually much more impressive than it seems at first glance. 71.2% puts it at 5th, which is 4 points below the leader (four points is a lot) and just over 1% lower than Anthropic’s own submission for Claude Sonnet 4 - the same model these guys are running. But the top rated submissions aren’t running production produc…
This is classic Goodhart's law. "When a measure becomes a target, it ceases to be a good measure"
Re: Qodo CLI agent scores 71.2% on SWE-bench Verified
#9I’ve been running a bunch of coding agents on benchmarks recently as part of consulting, and this is actually much more impressive than it seems at first glance. 71.2% puts it at 5th, which is 4 points below the leader (four points is a lot) and just over 1% lower than Anthropic’s own submission for Claude Sonnet 4 - the same model these guys are running. But the top rated submissions aren’t running production produc…
I think multiple attempts are completely understandable and even expected? How is that defeating the purpose of the benchmark?
Re: Qodo CLI agent scores 71.2% on SWE-bench Verified
#10I’ve been running a bunch of coding agents on benchmarks recently as part of consulting, and this is actually much more impressive than it seems at first glance. 71.2% puts it at 5th, which is 4 points below the leader (four points is a lot) and just over 1% lower than Anthropic’s own submission for Claude Sonnet 4 - the same model these guys are running. But the top rated submissions aren’t running production produc…