I've been using Warp for the past few weeks and it's been incredibly impressive over other agentic coding services/platforms. Curious how Qodo stacks up.
Qodo CLI agent scores 71.2% on SWE-bench Verified
11–20 of 60 posts
Re: Qodo CLI agent scores 71.2% on SWE-bench Verified
#12I’ve been running a bunch of coding agents on benchmarks recently as part of consulting, and this is actually much more impressive than it seems at first glance. 71.2% puts it at 5th, which is 4 points below the leader (four points is a lot) and just over 1% lower than Anthropic’s own submission for Claude Sonnet 4 - the same model these guys are running. But the top rated submissions aren’t running production produc…
I think multiple attempts are completely understandable and even expected? How is that defeating the purpose of the benchmark?
It’s a pass@1 benchmark. When submitting you need to check a box that there was only 1 attempt per problem. See here for example:
https://github.com/SWE-bench/experiments/pull/219
Building multiple attempts into your agent is stretching the rules, even if technically it’s acceptable
Re: Qodo CLI agent scores 71.2% on SWE-bench Verified
#13I’ve been running a bunch of coding agents on benchmarks recently as part of consulting, and this is actually much more impressive than it seems at first glance. 71.2% puts it at 5th, which is 4 points below the leader (four points is a lot) and just over 1% lower than Anthropic’s own submission for Claude Sonnet 4 - the same model these guys are running. But the top rated submissions aren’t running production produc…
Finally someone mentions Refact, I was in contact with the team, rooting for them really.
Re: Qodo CLI agent scores 71.2% on SWE-bench Verified
#14I've been using Warp for the past few weeks and it's been incredibly impressive over other agentic coding services/platforms. Curious how Qodo stacks up.
When I tried warp I was convinced that was where the industry was going (agents as terminal replacement), but it felt a bit too heavy to me so I haven’t been using it lately. Still think all things will converge on terminal and browser replacement.
Re: Qodo CLI agent scores 71.2% on SWE-bench Verified
#15So this is from the same company that wrote a blog post with sentences that don't even make sense:
https://news.ycombinator.com/item?id=44833929, my comment https://news.ycombinator.com/item?id=44835939
Re: Qodo CLI agent scores 71.2% on SWE-bench Verified
#16I’ve been running a bunch of coding agents on benchmarks recently as part of consulting, and this is actually much more impressive than it seems at first glance. 71.2% puts it at 5th, which is 4 points below the leader (four points is a lot) and just over 1% lower than Anthropic’s own submission for Claude Sonnet 4 - the same model these guys are running. But the top rated submissions aren’t running production produc…
Is there something in this multi-agent approach that makes the setup more specific to just the test at hand and less general to real engineering tasks? If not, then this multi-agent system will just become what you get out of the box in a future product. Multiple attempts per problem (as long as there's no human intervention or selection between them) is a perfectly fine approach for agents because that's not an issue from the perspective of an engineer using the product. A single agent is already a multi-step usage of LLMs and it sounds like this is just another meta level of that.
Re: Qodo CLI agent scores 71.2% on SWE-bench Verified
#17I’ve been running a bunch of coding agents on benchmarks recently as part of consulting, and this is actually much more impressive than it seems at first glance. 71.2% puts it at 5th, which is 4 points below the leader (four points is a lot) and just over 1% lower than Anthropic’s own submission for Claude Sonnet 4 - the same model these guys are running. But the top rated submissions aren’t running production produc…
One thing with SWE bench is making sure there's zero leakage of information into the LLM context.
I.e. the agent cannot even know which tests are failing.
It has to both fix the issue based just on the issue text and fix it in the specific way the unit test, which it cannot see, expects.
For this reason I find the benchmark a little disconnected from the reality of software engineering.
Re: Qodo CLI agent scores 71.2% on SWE-bench Verified
#18We need some international body to start running these tests… I just can’t trust these numbers any longer. We need a platform for this, something at least we can get some peer reviews
Re: Qodo CLI agent scores 71.2% on SWE-bench Verified
#19Earlier quoted context omitted.
I think multiple attempts are completely understandable and even expected? How is that defeating the purpose of the benchmark?
It’s a pass@1 benchmark. When submitting you need to check a box that there was only 1 attempt per problem. See here for example: https://github.com/SWE-bench/experiments/pull/219 Building multiple attempts into your agent is stretching the rules, even if technically it’s acceptable
From my perspective as a potential user the number of attempts is the number of times I have to tell it what to do. If you have an agent that makes a single attempt and is 60% accurate vs another that makes 5 attempts and is 80% accurate, why would you care that each individual attempt of the 2nd model is less accurate than the first?
Re: Qodo CLI agent scores 71.2% on SWE-bench Verified
#20Earlier quoted context omitted.
It’s a pass@1 benchmark. When submitting you need to check a box that there was only 1 attempt per problem. See here for example: https://github.com/SWE-bench/experiments/pull/219 Building multiple attempts into your agent is stretching the rules, even if technically it’s acceptable
From my perspective as a potential user the number of attempts is the number of times I have to tell it what to do. If you have an agent that makes a single attempt and is 60% accurate vs another that makes 5 attempts and is 80% accurate, why would you care that each individual attempt of the 2nd model is less accurate than the first?
I think it depends on "But the top rated submissions aren’t running production products" It sounds like they're shipping a product without the debug agent/try-again logic, and that's just for the benchmark, so you wouldn't get the performance they get as a user.