Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qodo CLI agent scores 71.2% on SWE-bench Verified

qodo.ai

Post reply on HN
Print view View original

31–40 of 60 posts

Re: Qodo CLI agent scores 71.2% on SWE-bench Verified

#31

Post by whymauri »

energy123 wrote: post #4
gronky_ wrote: post #2

I’ve been running a bunch of coding agents on benchmarks recently as part of consulting, and this is actually much more impressive than it seems at first glance. 71.2% puts it at 5th, which is 4 points below the leader (four points is a lot) and just over 1% lower than Anthropic’s own submission for Claude Sonnet 4 - the same model these guys are running. But the top rated submissions aren’t running production produc…

What are the typical context lengths in SWE-bench problems? Does it partly measure performance in the 64-128k context range?

This is what the rows look like:

https://huggingface.co/datasets/princeton-nlp/SWE-bench_Veri...

Its up to your retrieval system/model to selectively hunt for relevant context. Here's a few critiques of the benchy:

https://x.com/brhydon/status/1953648884309536958

Re: Qodo CLI agent scores 71.2% on SWE-bench Verified

#32

Post by whymauri »

terminalshort wrote: post #19
gronky_ wrote: post #12

Earlier quoted context omitted.

It’s a pass@1 benchmark. When submitting you need to check a box that there was only 1 attempt per problem. See here for example: https://github.com/SWE-bench/experiments/pull/219 Building multiple attempts into your agent is stretching the rules, even if technically it’s acceptable

From my perspective as a potential user the number of attempts is the number of times I have to tell it what to do. If you have an agent that makes a single attempt and is 60% accurate vs another that makes 5 attempts and is 80% accurate, why would you care that each individual attempt of the 2nd model is less accurate than the first?

Papers have been doing rollouts that involve a model proposing N solutions and then self-reviewing to choose the best one (prior to the verifier). So far, I think that's been counted as one pass.

Re: Qodo CLI agent scores 71.2% on SWE-bench Verified

#33

Post by whymauri »

itamarcode wrote: post #29

Unlike most SWE bench submissions, Qodo Command one uses the product directly. I think that the next step is getting an official "checked" mark by the SWE bench team

I feel like the bash only SWE Bench Verified (a.k.a model + mini-swe-agent) is the closest thing to measuring the inherent ability of the model vs. the scaffolding.

https://github.com/SWE-agent/mini-swe-agent

Re: Qodo CLI agent scores 71.2% on SWE-bench Verified

#35

Post by radarsat1 »

terminalshort wrote: post #19
gronky_ wrote: post #12

Earlier quoted context omitted.

It’s a pass@1 benchmark. When submitting you need to check a box that there was only 1 attempt per problem. See here for example: https://github.com/SWE-bench/experiments/pull/219 Building multiple attempts into your agent is stretching the rules, even if technically it’s acceptable

From my perspective as a potential user the number of attempts is the number of times I have to tell it what to do. If you have an agent that makes a single attempt and is 60% accurate vs another that makes 5 attempts and is 80% accurate, why would you care that each individual attempt of the 2nd model is less accurate than the first?

I was thinking about this recently with respect to how many agent systems now let you specify a smaller/faster model for easier tasks and a bigger model for harder tasks.

It's interesting to think about what the trade-offs are. Assuming the system can properly classify a task as easy or hard (big "if" but I guess there are ways), there is nonetheless more to think about, depending on your pricing plan.

For subscription pricing, I guess you don't really care which model runs and in fact it's hard to find a reason to ever run the smaller model, so choosing between the models is more in the provider's interests for cost efficiency.

But for pay-per-use pricing, But if you have a bigger model that can get the answer right 80% of the time, and a smaller model that can handle smaller changes and get things right 60% of the time but correct its mistakes, then the system should try to run it on as many tasks as possible to save you money.. but in the end if ends up having to make a lot of corrections, then maybe you end up needing more total requests than the larger model. In that case maybe it's actually cheaper to run the larger model, if it takes fewer requests.

So I wonder how that kind of trade-off could be effectively calculated. I guess if you can figure out when "retries" happen you can count them and do some statistics on which model is more likely to work out in fewer shots. It's pretty complicated though, when you start to think about it in detail.

I do wonder if even having BOTH the smaller and bigger model make hypotheses, and try the smaller model's idea first, then if it fails, try the bigger model's idea, might be the way to go.

Re: Qodo CLI agent scores 71.2% on SWE-bench Verified

#36

Post by bluelightning2k »

Roritharr wrote: post #13
gronky_ wrote: post #2

I’ve been running a bunch of coding agents on benchmarks recently as part of consulting, and this is actually much more impressive than it seems at first glance. 71.2% puts it at 5th, which is 4 points below the leader (four points is a lot) and just over 1% lower than Anthropic’s own submission for Claude Sonnet 4 - the same model these guys are running. But the top rated submissions aren’t running production produc…

Finally someone mentions Refact, I was in contact with the team, rooting for them really.

Just looked them up. Their pricing is around buying "coins" with no transparency as to what that gets. Hard pass

Re: Qodo CLI agent scores 71.2% on SWE-bench Verified

#40

Post by jasonjmcghee »

ambicapter wrote: post #21
thinkingtoilet wrote: post #8

Earlier quoted context omitted.

This is classic Goodhart's law. "When a measure becomes a target, it ceases to be a good measure" https://en.wikipedia.org/wiki/Goodhart%27s_law

It's really not that hard to not build a custom bench setup to game the benchmark instead of just using your product straight out of the box, though.

Right. Building a custom setup is blatant- that will wildly overfit.

But let's say a group uses it as a metric as part of CI and each new idea / feature they create runs against SWE bench. Maybe they have parameterized bits and pieces they adjust, maybe they have multiple candidates datasets for fine tuning, maybe they're choosing between checkpoints.

This will also end up overfitting - especially if done habitually. It might be a great metric and result in a more powerful overall model. Or it might not.

Post reply on HN