Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Optio – Orchestrate AI coding agents in K8s to go from ticket to PR

github.com

Post reply on HN
Print view View original

61–70 of 77 posts

Re: Show HN: Optio – Orchestrate AI coding agents in K8s to go from ticket to PR

#61

Post by vidarh »

pianopatrick wrote: post #24

I wonder, based on your experience, how hard would it be to improve your system to have an AI agent review the software and suggest tickets? Like, can an AI agent use a browser, attempt to use the software, find bugs and create a ticket? Can an AI agent use a browser, try to use the software and suggest new features?

Yes, they can, and they do a reasonably good job at it. Hand them playwright or similar, and point them at it. The caveat is that they're often "lazy", and it takes some practice to coax them into being thorough (hot tip: have one write a list of things to probe and test, and tell it to use sub agents to address each; otherwise they tend to decide very quickly it's too tedious and start taking shortcuts)

Re: Show HN: Optio – Orchestrate AI coding agents in K8s to go from ticket to PR

#62

Post by stingraycharles »

vidarh wrote: post #58
stingraycharles wrote: post #34

I’ve come to the realization that these kind of systems don’t work, and that a human in the loop is crucial for task planning; the LLM’s role being to identify issues, communicate the design / architecture, etc before it’s handed off, otherwise the LLM always ends up doing not entirely the correct thing. How is this part tackled when all that you have is GH issues? Doesn’t this work only for the most trivial issues?

I've come to the opposite conclusions: The big limitation of systems like this is starting and ending with human involvement at the same level, instead of directing at a higher level. You end up quibbling over detail the agents can handle themselves with sufficient guardrails and process, instead of setting higher level requirements and reviewing higher level decisions and outcomes, and dealing with exceptions. You c…

I find that that doesn’t work in the long run. Software agents are not yet capable of maintaining a decently active repository for extended periods of time.

I am all for delegating everything to AI agents, but it just becomes a mess over time if you don’t steer things often enough.

Re: Show HN: Optio – Orchestrate AI coding agents in K8s to go from ticket to PR

#63

Post by SR2Z »

59nadir wrote: post #23
SR2Z wrote: post #21

Earlier quoted context omitted.

I mean, having unit tests and not allowing PRs in unless they all pass is pretty easy (or requiring human review to remove a test!). A software engineer takes a spec which "shifts the distribution of acceptable responses" for their output. If they're 100% accurate ( snort ), how good does an LLM have to be for you to accept its review as reasonable?

We've seen public examples of where LLMs literally disable or remove tests in order to pass. I'm not sure having tests and asking LLMs to not merge things before passing them being "easy" matters much when the failure modes here are so plentiful and broad in nature.

So... add another presubmit test that fails when a test is removed. Require human reviews.

It's not like a human being always pushes correct code, my risk assessment for an LLM reading a small bug and just making a PR is that thinking too hard is a waste of time. My risk assessment for a human is very similar, because actually catching issues during code review is best done by tests anyways. If the tests can't tell you if your code is good or not then it really doesn't matter if it's a human or an LLM, you're mostly just guessing if things are going to work and you WILL push bad code that gets caught in prod.

Re: Show HN: Optio – Orchestrate AI coding agents in K8s to go from ticket to PR

#64

Post by vidarh »

stingraycharles wrote: post #62
vidarh wrote: post #58

Earlier quoted context omitted.

I've come to the opposite conclusions: The big limitation of systems like this is starting and ending with human involvement at the same level, instead of directing at a higher level. You end up quibbling over detail the agents can handle themselves with sufficient guardrails and process, instead of setting higher level requirements and reviewing higher level decisions and outcomes, and dealing with exceptions. You c…

I find that that doesn’t work in the long run. Software agents are not yet capable of maintaining a decently active repository for extended periods of time. I am all for delegating everything to AI agents, but it just becomes a mess over time if you don’t steer things often enough.

Not my experience at all. If anything, they make it cheap enough to deal with tech debt that it is far easier to justify being strict.

EDIT: I'll add that you can't expect it to guess what you want, but you can let it manage how it delivers it. We don't expect e.g. a product manager to dictate how developers deliver the code, just what the acceptance criteria is, and that's where I'm headed.

Re: Show HN: Optio – Orchestrate AI coding agents in K8s to go from ticket to PR

#65

Post by pianopatrick »

ramon156 wrote: post #28
pianopatrick wrote: post #24

I wonder, based on your experience, how hard would it be to improve your system to have an AI agent review the software and suggest tickets? Like, can an AI agent use a browser, attempt to use the software, find bugs and create a ticket? Can an AI agent use a browser, try to use the software and suggest new features?

I think it's more important to pin down where a human must be in order for this not to become a mess. Or have we skipped that step entirely?

Personally my theory is that to solve the messiness we will need some new frameworks and even languages that are designed to catch AI mistakes in large code bases. For example, AIs in the past would sometimes hallucinate methods that do not exist. But in a language with a strong type system a static type checker should be able to catch that mistake and give the AI automated feedback to fix that mistake without a human in the loop.

As far as humans in the loop, the only human we ultimately cannot get rid of is the user. But I think with a combo of user feedback forms and automated metrics we can give AI a lot of feedback about how good software is just from users using the software.

Re: Show HN: Optio – Orchestrate AI coding agents in K8s to go from ticket to PR

#66

Post by berkay »

mshark wrote: post #37
stingraycharles wrote: post #34

I’ve come to the realization that these kind of systems don’t work, and that a human in the loop is crucial for task planning; the LLM’s role being to identify issues, communicate the design / architecture, etc before it’s handed off, otherwise the LLM always ends up doing not entirely the correct thing. How is this part tackled when all that you have is GH issues? Doesn’t this work only for the most trivial issues?

Had the same realization which inspired eforge (shameless plug) https://github.com/eforge-build/eforge - planning stays in the developer’s control with all engineering (agent orchestration) handed off to eforge. This has been working well for a solo or siloed developer (me) that is free to plan independently. Allows the developer to confidently stay in the planning plane while eforge handles the rest using a methodol…

I like the separation of planning and execution. I think the right set of artifacts to pass on to the execution will evolve but may be it's different for different types of work.

From the project: "The plugin enqueues the input and a daemon picks it up - planning, building, reviewing, and validating autonomously."

The part that is not clear to me (and causes most problems for me) is the "validating". It makes a mistake, or decides mocking an interface is fine, etc. declares success and moves on to the next. The bigger the project the more small mistakes compound. It sounds like the agent is doing the validation. What's the approach here for validation?

Re: Show HN: Optio – Orchestrate AI coding agents in K8s to go from ticket to PR

#67

Post by jordanedev »

nevon wrote: post #32
jawiggins wrote: post #6

Earlier quoted context omitted.

Oh good question, I haven't thought deeply about this. Right now nothing special happens, so claude/codex can access their normal tools and make web calls. I suppose that also means they could figure out they're running in a k8s pod and do service discovery and start calling things. What kind of features would you be interested in seeing around this? Maybe a toggle to disable internet connections or other connections…

Network policies controlling egress would be one thing. I haven't seen how you make secrets available to the agent, but I would imagine you would need to proxy calls through a mitm proxy to replace tokens with real secrets, or some other way to make sure the agent cannot access the secrets themselves. Specifically for an agent that works with code, I could imagine being able to run docker-in-docker will probably be r…

That's exactly what i did personnaly on my oss repo https://github.com/ysa-ai/ysa

I want to run my agents fully isolated with headless mode. To achieve that safely you have to run a proxy

Re: Show HN: Optio – Orchestrate AI coding agents in K8s to go from ticket to PR

#68

Post by Ryand1234 »

stingraycharles wrote: post #34

I’ve come to the realization that these kind of systems don’t work, and that a human in the loop is crucial for task planning; the LLM’s role being to identify issues, communicate the design / architecture, etc before it’s handed off, otherwise the LLM always ends up doing not entirely the correct thing. How is this part tackled when all that you have is GH issues? Doesn’t this work only for the most trivial issues?

I kind of agree. For features, we need human in the loop to verify what agent needs to build but for bugs and all, I feel these systems can work.

Re: Show HN: Optio – Orchestrate AI coding agents in K8s to go from ticket to PR

#69

Post by tim-projects »

vidarh wrote: post #58
stingraycharles wrote: post #34

I’ve come to the realization that these kind of systems don’t work, and that a human in the loop is crucial for task planning; the LLM’s role being to identify issues, communicate the design / architecture, etc before it’s handed off, otherwise the LLM always ends up doing not entirely the correct thing. How is this part tackled when all that you have is GH issues? Doesn’t this work only for the most trivial issues?

I've come to the opposite conclusions: The big limitation of systems like this is starting and ending with human involvement at the same level, instead of directing at a higher level. You end up quibbling over detail the agents can handle themselves with sufficient guardrails and process, instead of setting higher level requirements and reviewing higher level decisions and outcomes, and dealing with exceptions. You c…

I don't believe comments like this. Sure it did work for ten hours but if you didn't review it you will sooner or later when it breaks. And it will. I run the agents all day and that's what happens - they do stuff that is unwanted but that you aren't aware of.
Post reply on HN