Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI

arxiv.org

Post reply on HN
Print view View original

1–10 of 49 posts

Re: SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI

#4

Post by challengerVIE »

To me using agents daily, the long term vision with maintainability in mind really makes the difference between us humans and agents, I like the idea. However evaluating long term maintainability over an average of just 500 loc changes does not sound like long term maintainability being measured here

Re: SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI

#5

Post by KronisLV »

> The benchmark comprises 100 tasks, each corresponding on average to an evolution history spanning 233 days and 71 consecutive commits in a real-world code repository.

This seems like a really cool thing to benchmark! Technically it'd be possible to take GitHub repos that the AI orgs probably already have, cross-reference the code against the issues and regressions, and train/validate on that.

The dataset would need to be way bigger to get close to the likes of SWE-bench: https://www.swebench.com/original.html

"Vibe coded stuff gets hard to maintain and will end up buggy." Yeah, so make models that deal with that better, optimize for maintainability and consistency.

Cool to see Claude doing decently though!

Re: SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI

#6

Post by woadwarrior01 »

Interesting benchmark.

I can't help but notice that they're benchmarking Opus 4.6 (Anthropic's latest and greatest model) against GPT-5.2 (which is three generations behind OpenAI's latest coding models: GPT-5.2-Codex, GPT-5.3-Codex and the latest GPT-5.4).

Re: SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI

#7

Post by aurareturn »

woadwarrior01 wrote: post #6

Interesting benchmark. I can't help but notice that they're benchmarking Opus 4.6 (Anthropic's latest and greatest model) against GPT-5.2 (which is three generations behind OpenAI's latest coding models: GPT-5.2-Codex, GPT-5.3-Codex and the latest GPT-5.4).

As far as I know, OpenAI did not release 5.3 Codex in their API. You can only use it with Codex CLI or app.

Re: SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI

#8

Post by woadwarrior01 »

KronisLV wrote: post #5

> The benchmark comprises 100 tasks, each corresponding on average to an evolution history spanning 233 days and 71 consecutive commits in a real-world code repository. This seems like a really cool thing to benchmark! Technically it'd be possible to take GitHub repos that the AI orgs probably already have, cross-reference the code against the issues and regressions, and train/validate on that. The dataset would need…

> Cool to see Claude doing decently though!

The scales do seem to be tipped in its favor (cf: my other comment in this thread).

Re: SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI

#9

Post by mentalgear »

Claude wins by a large margin

* Claude Opus 4.6 : 0.71

* Claude Opus 4.5 : 0.51

* KIMI-K2.5 : 0.37

* GLM-5 : 0.36

* GPT-5.2 : 0.23

Note: later GPT versions seem to be only available within openAi's proprietary codex cli, so can't be tested - and if tested via the codex cli "harness" it wouldn't be a pure model-to-model comparison any more.

---

Of course, the interesting follow-up question is: How well perform these models with added agent tooling ("harness") ?

Maybe someone has tokens to burn and can run a matrix of agent tools over the top models and provide the results?

Post reply on HN