Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI

arxiv.org

Post reply on HN
Print view View original

21–30 of 49 posts

Re: SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI

#21

Post by jsemrau »

mentalgear wrote: post #9

Claude wins by a large margin * Claude Opus 4.6 : 0.71 * Claude Opus 4.5 : 0.51 * KIMI-K2.5 : 0.37 * GLM-5 : 0.36 * GPT-5.2 : 0.23 Note: later GPT versions seem to be only available within openAi's proprietary codex cli, so can't be tested - and if tested via the codex cli "harness" it wouldn't be a pure model-to-model comparison any more. --- Of course, the interesting follow-up question is: How well perform these m…

I reached the same conclusion. I tried using both for my personal investment ambient using agent-pair programming to build and agentic intelligence layer for stocks and the difference between the 2 models if astounding.

Re: SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI

#25

Post by jlebensold »

coder_decoder wrote: post #24

[flagged]

I've been building a similar loop with jetty.io for the last few months exclusively focused on data science workflows. I think that there's a lot of hill-climbing that can be accomplished by having a clear runbook.

Re: SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI

#28

Post by andai »

mentalgear wrote: post #9

Claude wins by a large margin * Claude Opus 4.6 : 0.71 * Claude Opus 4.5 : 0.51 * KIMI-K2.5 : 0.37 * GLM-5 : 0.36 * GPT-5.2 : 0.23 Note: later GPT versions seem to be only available within openAi's proprietary codex cli, so can't be tested - and if tested via the codex cli "harness" it wouldn't be a pure model-to-model comparison any more. --- Of course, the interesting follow-up question is: How well perform these m…

>if tested via the codex cli "harness" it wouldn't be a pure model-to-model comparison any more.

Well that's already not a very fair comparison, we've known for years (one of the early-ish LLM papers, maybe someone knows which one) that prompting makes an enormous difference on agent performance, and most strikingly, the same prompt that massively boosts performance on one model, can massively reduce performance on another.

So you already need to fine-tune the prompts for the model, if you want anything approaching best results.

Now what's really amusing is that if you run models without their official harness, they can actually do way better on some benchmarks! [0] e.g. On Terminal Bench 2, Claude Opus 4.6 goes from #33 (Claude Code) to #5 (custom harness). Similar results for Codex.

Now, this is "for this one very specific benchmark", but I still thought it was funny, since you'd expect "the harness made by the same company" to be the best for all tasks, but that's clearly not the case. (For specific tasks, it's actually quite trivial to outperform a general purpose harness.)

[0] https://www.tbench.ai/leaderboard/terminal-bench/2.0

Re: SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI

#29

Post by climike »

mentalgear wrote: post #9

Claude wins by a large margin * Claude Opus 4.6 : 0.71 * Claude Opus 4.5 : 0.51 * KIMI-K2.5 : 0.37 * GLM-5 : 0.36 * GPT-5.2 : 0.23 Note: later GPT versions seem to be only available within openAi's proprietary codex cli, so can't be tested - and if tested via the codex cli "harness" it wouldn't be a pure model-to-model comparison any more. --- Of course, the interesting follow-up question is: How well perform these m…

We are working on supporting agent harnesses @ www.cliwatch.com, so both 1. LLM model as well 2. LLM model + harness performance can be evaluated against your software/CLI. We also support building evals against your doc suite. End result is that you’ll feel more comfortable shipping CLIs that work for your agentic users!:)

Re: SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI

#30

Post by jasonjmcghee »

gizmodo59 wrote: post #16

Unfortunately the paper doesn’t include gpt 5.3 which was released around the same time as opus 4.6 and also gpt 5.4 few days back. Both are available via api https://developers.openai.com/api/docs/models/gpt-5.3-codex IMHO The harness must be used when running these experiments. The model vendors know best on giving the best harness with gpt 5.4 and codex or Claude code with opus 4.6 which makes a big difference if…

> model vendors know best on giving the best harness

This was only true for Claude Code for a while. Codex was poor and Gemini was unusable.

Since then Codex has gotten quite good.

Post reply on HN