Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI

arxiv.org

Post reply on HN
Print view View original

41–49 of 49 posts

Re: SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI

#43

Post by saltpath »

agent5ravi wrote: post #38

[flagged]

CI pass/fail captures regression, but there's a layer beneath it that benchmarks can't touch: what exactly did the agent submit to each external API, and can you prove it after the fact? In the benchmark context this doesn't matter everything runs locally. In production it does. The agent calls a third-party service at 2am, the service claims it returned an error, your agent retried and billed you twice. Your logs say one thing, their logs say another. The integrity problem isn't just "did the code work" it's "what was the exact request/response pair, timestamped, by whom, provably." CI solves the first. Something else has to solve the second.

Re: SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI

#45

Post by jsemrau »

jasonjmcghee wrote: post #30
gizmodo59 wrote: post #16

Unfortunately the paper doesn’t include gpt 5.3 which was released around the same time as opus 4.6 and also gpt 5.4 few days back. Both are available via api https://developers.openai.com/api/docs/models/gpt-5.3-codex IMHO The harness must be used when running these experiments. The model vendors know best on giving the best harness with gpt 5.4 and codex or Claude code with opus 4.6 which makes a big difference if…

> model vendors know best on giving the best harness This was only true for Claude Code for a while. Codex was poor and Gemini was unusable. Since then Codex has gotten quite good.

It still fubars my code regularly at 11x the price. Github Copilot Agentic Mode + Sonnet 4.6 is stable and inexpensive.

Re: SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI

#46

Post by noobcoder »

Passing tests doesn’t mean you have a working codebase. Benchmarks that rely on a fixed test suite create a real optimization problem agents (or/and even humans) learn to satisfy the tests rather than preserve the deeper properties that make the system maintainable. AI write test cases which it thinks is easier for it to satisfy and not adhere-ing to business logic

We see this firsthand at Prismor with auto generated security fixes. Even with the best LLMs, validating fixes is the real bottleneck our pipeline struggles to exceed 70% on an internal golden dataset (which itself is somewhat biased).

Many patches technically fix the vulnerability but introduce semantic regressions or architectural drift. Passing tests is a weak signal and proving a fix is truly safe to merge is much harder

Re: SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI

#47

Post by mike_hearn »

pizlonator wrote: post #33
mike_hearn wrote: post #14

Earlier quoted context omitted.

It's the other way around - Claude Code is the proprietary one. Codex CLI is open source: https://github.com/openai/codex You can definitely access the latest models via the API. That's how Codex CLI works.

gpt-5.3 was not accessible via API, at least for me But it was in codex

Ah ha, I know what you're hitting.

5.3-codex is only available via the Responses API, not the Completions API. Two different APIs for model access. If you were using Completions you have to port to Responses. It's not that hard. I did this for my own agent the other week. I think it might be like that for all their new models from now on. Responses is a much more powerful API. It's more like a front to ChatGPT than the underlying models.

Re: SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI

#48

Post by pizlonator »

mike_hearn wrote: post #47
pizlonator wrote: post #33

Earlier quoted context omitted.

gpt-5.3 was not accessible via API, at least for me But it was in codex

Ah ha, I know what you're hitting. 5.3-codex is only available via the Responses API, not the Completions API. Two different APIs for model access. If you were using Completions you have to port to Responses. It's not that hard. I did this for my own agent the other week. I think it might be like that for all their new models from now on. Responses is a much more powerful API. It's more like a front to ChatGPT than t…

I am using responses API (these days I only use responses API when talking to OpenAI)

I'll try again, maybe I was doing something else wrong, or something changed

Re: SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI

#49

Post by mike_hearn »

pizlonator wrote: post #48
mike_hearn wrote: post #47

Earlier quoted context omitted.

Ah ha, I know what you're hitting. 5.3-codex is only available via the Responses API, not the Completions API. Two different APIs for model access. If you were using Completions you have to port to Responses. It's not that hard. I did this for my own agent the other week. I think it might be like that for all their new models from now on. Responses is a much more powerful API. It's more like a front to ChatGPT than t…

I am using responses API (these days I only use responses API when talking to OpenAI) I'll try again, maybe I was doing something else wrong, or something changed

OK, in that case I'm not sure. But Codex can't hide anything. I don't think there are special codes compiled into the binaries or anything like that.
Post reply on HN