Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI

arxiv.org

Post reply on HN
Print view View original

11–20 of 49 posts

Re: SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI

#11

Post by re-thc »

woadwarrior01 wrote: post #6

Interesting benchmark. I can't help but notice that they're benchmarking Opus 4.6 (Anthropic's latest and greatest model) against GPT-5.2 (which is three generations behind OpenAI's latest coding models: GPT-5.2-Codex, GPT-5.3-Codex and the latest GPT-5.4).

5.2 and 5.2 Codex is arguably the same gen.

Re: SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI

#14

Post by mike_hearn »

mentalgear wrote: post #9

Claude wins by a large margin * Claude Opus 4.6 : 0.71 * Claude Opus 4.5 : 0.51 * KIMI-K2.5 : 0.37 * GLM-5 : 0.36 * GPT-5.2 : 0.23 Note: later GPT versions seem to be only available within openAi's proprietary codex cli, so can't be tested - and if tested via the codex cli "harness" it wouldn't be a pure model-to-model comparison any more. --- Of course, the interesting follow-up question is: How well perform these m…

It's the other way around - Claude Code is the proprietary one. Codex CLI is open source:

https://github.com/openai/codex

You can definitely access the latest models via the API. That's how Codex CLI works.

Re: SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI

#15

Post by baalimago »

aurareturn wrote: post #7
woadwarrior01 wrote: post #6

Interesting benchmark. I can't help but notice that they're benchmarking Opus 4.6 (Anthropic's latest and greatest model) against GPT-5.2 (which is three generations behind OpenAI's latest coding models: GPT-5.2-Codex, GPT-5.3-Codex and the latest GPT-5.4).

As far as I know, OpenAI did not release 5.3 Codex in their API. You can only use it with Codex CLI or app.

It's there, you just need to use it with the responses API. Set model field to 'gpt-5.3-codex'

Re: SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI

#16

Post by gizmodo59 »

Unfortunately the paper doesn’t include gpt 5.3 which was released around the same time as opus 4.6 and also gpt 5.4 few days back. Both are available via api

https://developers.openai.com/api/docs/models/gpt-5.3-codex

IMHO The harness must be used when running these experiments. The model vendors know best on giving the best harness with gpt 5.4 and codex or Claude code with opus 4.6 which makes a big difference if you are running any kind of agentic coding tasks.

I see both Claude and gpt to be neck and neck in coding. Every other model+harness is definitely 3-6 months behind. Right now codex seems to be the best in terms of solving complex bugs, long running tasks, much higher limits and even speed while Claude seems to do well in front end and their cli ux seems nice! Codex app is very good though (wish it wasn’t electron as a memory hog but it’s good)

Re: SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI

#17

Post by KronisLV »

baalimago wrote: post #13

Replace "Agent" with "Employee" and apply the same algorithm. Evaluate employee efficiency. Profit?

I'd unironically (and privately) want to do that with the code of both myself and those around me - to maybe see who I should listen more to, as well as who maybe less (ideally down to the feature level), because everyone has opinions, sometimes loud ones, but some approaches lead to a lot of churn and issues over the years.

Re: SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI

#18

Post by p1esk »

gizmodo59 wrote: post #16

Unfortunately the paper doesn’t include gpt 5.3 which was released around the same time as opus 4.6 and also gpt 5.4 few days back. Both are available via api https://developers.openai.com/api/docs/models/gpt-5.3-codex IMHO The harness must be used when running these experiments. The model vendors know best on giving the best harness with gpt 5.4 and codex or Claude code with opus 4.6 which makes a big difference if…

Are you saying they did not use native harnesses like Claude Code or Codex? How did they do it then?
Post reply on HN