Qodo CLI agent scores 71.2% on SWE-bench Verified

Post by ambicapter » Tue, Aug 12, 2025, 1:21 PM UTC

thinkingtoilet wrote: post #8

gronky_ wrote: post #2

I’ve been running a bunch of coding agents on benchmarks recently as part of consulting, and this is actually much more impressive than it seems at first glance. 71.2% puts it at 5th, which is 4 points below the leader (four points is a lot) and just over 1% lower than Anthropic’s own submission for Claude Sonnet 4 - the same model these guys are running. But the top rated submissions aren’t running production produc…

This is classic Goodhart's law. "When a measure becomes a target, it ceases to be a good measure" https://en.wikipedia.org/wiki/Goodhart%27s_law

Post by clutchdude » Tue, Aug 12, 2025, 1:26 PM UTC

thinkingtoilet wrote: post #8

gronky_ wrote: post #2

I’ve been running a bunch of coding agents on benchmarks recently as part of consulting, and this is actually much more impressive than it seems at first glance. 71.2% puts it at 5th, which is 4 points below the leader (four points is a lot) and just over 1% lower than Anthropic’s own submission for Claude Sonnet 4 - the same model these guys are running. But the top rated submissions aren’t running production produc…

This is classic Goodhart's law. "When a measure becomes a target, it ceases to be a good measure" https://en.wikipedia.org/wiki/Goodhart%27s_law

Post by gronky_ » Tue, Aug 12, 2025, 1:29 PM UTC

terminalshort wrote: post #19

gronky_ wrote: post #12

Earlier quoted context omitted.

It’s a pass@1 benchmark. When submitting you need to check a box that there was only 1 attempt per problem. See here for example: https://github.com/SWE-bench/experiments/pull/219 Building multiple attempts into your agent is stretching the rules, even if technically it’s acceptable

From my perspective as a potential user the number of attempts is the number of times I have to tell it what to do. If you have an agent that makes a single attempt and is 60% accurate vs another that makes 5 attempts and is 80% accurate, why would you care that each individual attempt of the 2nd model is less accurate than the first?

    def retry_agent(problem):

        for attempt in range(n):

            result = agent(problem)

            if result.success:

                return result

        return result

    return retry_agent

Post by gronky_ » Tue, Aug 12, 2025, 1:37 PM UTC

gronky_ wrote: post #25

terminalshort wrote: post #19

Earlier quoted context omitted.

From my perspective as a potential user the number of attempts is the number of times I have to tell it what to do. If you have an agent that makes a single attempt and is 60% accurate vs another that makes 5 attempts and is 80% accurate, why would you care that each individual attempt of the 2nd model is less accurate than the first?

This ok from your perspective then? def make_pass@1_agent(agent, n): def retry_agent(problem): for attempt in range(n): result = agent(problem) if result.success: return result return result return retry_agent

Post by itamarcode » Tue, Aug 12, 2025, 2:06 PM UTC