Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Optio – Orchestrate AI coding agents in K8s to go from ticket to PR

github.com

Post reply on HN
Print view View original

51–60 of 77 posts

Re: Show HN: Optio – Orchestrate AI coding agents in K8s to go from ticket to PR

#51

Post by jawiggins »

stingraycharles wrote: post #34

I’ve come to the realization that these kind of systems don’t work, and that a human in the loop is crucial for task planning; the LLM’s role being to identify issues, communicate the design / architecture, etc before it’s handed off, otherwise the LLM always ends up doing not entirely the correct thing. How is this part tackled when all that you have is GH issues? Doesn’t this work only for the most trivial issues?

Maybe - I do think as the model get better they'll be able to handle more and more difficult tasks. And yet, even if they can only solve the simplest issues now, why not let them so you can focus on the more important things?

Re: Show HN: Optio – Orchestrate AI coding agents in K8s to go from ticket to PR

#52

Post by jawiggins »

maxdo wrote: post #43

Is the pod per repo or per task ?

One pod is an instance of a repo, you can set the number of instances of each agent/task that can be running on a pod at a time. For >1, each agent should be using it's own worktree.

Re: Show HN: Optio – Orchestrate AI coding agents in K8s to go from ticket to PR

#53

Post by [deleted] »

SR2Z wrote: post #21
kristjansson wrote: post #14

Earlier quoted context omitted.

> to make sure you've really got to be careful with absolute language like this in reference to LLMs. A review agent provides no guarantees whatsoever, just shifts the distribution of acceptable responses, hopefully in a direction the user prefers.

I mean, having unit tests and not allowing PRs in unless they all pass is pretty easy (or requiring human review to remove a test!). A software engineer takes a spec which "shifts the distribution of acceptable responses" for their output. If they're 100% accurate ( snort ), how good does an LLM have to be for you to accept its review as reasonable?

[deleted]

Re: Show HN: Optio – Orchestrate AI coding agents in K8s to go from ticket to PR

#56

Post by jawiggins »

59nadir wrote: post #23
SR2Z wrote: post #21

Earlier quoted context omitted.

I mean, having unit tests and not allowing PRs in unless they all pass is pretty easy (or requiring human review to remove a test!). A software engineer takes a spec which "shifts the distribution of acceptable responses" for their output. If they're 100% accurate ( snort ), how good does an LLM have to be for you to accept its review as reasonable?

We've seen public examples of where LLMs literally disable or remove tests in order to pass. I'm not sure having tests and asking LLMs to not merge things before passing them being "easy" matters much when the failure modes here are so plentiful and broad in nature.

You'd want to have the tests run as a github action and then fail the check if the tests don't pass. Optio will resume agents when the actions fail and tell them to fix the failures.

Re: Show HN: Optio – Orchestrate AI coding agents in K8s to go from ticket to PR

#57

Post by fhouser »

jawiggins wrote: post #55
fhouser wrote: post #47

Hot take: You should want to review your agents' output and progress.

Yeah totally, you don't have to auto-merge anything - you can review the PRs yourself

Yeah, I think that's the most important part in these new types of processes. Although it is tempting to just let an agent run with it for a while.

Re: Show HN: Optio – Orchestrate AI coding agents in K8s to go from ticket to PR

#58

Post by vidarh »

stingraycharles wrote: post #34

I’ve come to the realization that these kind of systems don’t work, and that a human in the loop is crucial for task planning; the LLM’s role being to identify issues, communicate the design / architecture, etc before it’s handed off, otherwise the LLM always ends up doing not entirely the correct thing. How is this part tackled when all that you have is GH issues? Doesn’t this work only for the most trivial issues?

I've come to the opposite conclusions: The big limitation of systems like this is starting and ending with human involvement at the same level, instead of directing at a higher level. You end up quibbling over detail the agents can handle themselves with sufficient guardrails and process, instead of setting higher level requirements and reviewing higher level decisions and outcomes, and dealing with exceptions.

You can afford a lot of extra guardrails and process to ensure sufficient quality when the result is a system that gets improved autonomously 24/7.

I'm on my way home from a client, and meanwhile another project has spent the last 10 hours improving with no involvement from me. I spent a few minutes reviewing things this morning, after it's spent the whole night improving unattended.

Re: Show HN: Optio – Orchestrate AI coding agents in K8s to go from ticket to PR

#59

Post by vidarh »

saltpath wrote: post #36

The parallel execution model makes sense for independent tickets but I'm wondering what happens when agent A is halfway through a PR touching shared/utils.py and agent B gets assigned a ticket that needs the same file. Does the orchestrator do any upfront dependency analysis to detect that, or do you just let them both run and deal with the conflict at merge time?

It's generally not worth it worrying about it too much other than at a very high level vs. letting them fight it out, as long as your test suite is good enough and your orchestrator is even moderately prepared to handle retries.
Post reply on HN