31–40 of 77 posts
I wonder, based on your experience, how hard would it be to improve your system to have an AI agent review the software and suggest tickets? Like, can an AI agent use a browser, attempt to use the software, find bugs and create a ticket? Can an AI agent use a browser, try to use the software and suggest new features?
Looks cool, congrats on the launch. Is there any sandbox isolation from the k8s platform layer? Wondering if this is suitable for multiple tenants or customers.
Oh good question, I haven't thought deeply about this. Right now nothing special happens, so claude/codex can access their normal tools and make web calls. I suppose that also means they could figure out they're running in a k8s pod and do service discovery and start calling things. What kind of features would you be interested in seeing around this? Maybe a toggle to disable internet connections or other connections…
Earlier quoted context omitted.
I mean, having unit tests and not allowing PRs in unless they all pass is pretty easy (or requiring human review to remove a test!). A software engineer takes a spec which "shifts the distribution of acceptable responses" for their output. If they're 100% accurate ( snort ), how good does an LLM have to be for you to accept its review as reasonable?
We've seen public examples of where LLMs literally disable or remove tests in order to pass. I'm not sure having tests and asking LLMs to not merge things before passing them being "easy" matters much when the failure modes here are so plentiful and broad in nature.
How is this part tackled when all that you have is GH issues? Doesn’t this work only for the most trivial issues?
I wonder, based on your experience, how hard would it be to improve your system to have an AI agent review the software and suggest tickets? Like, can an AI agent use a browser, attempt to use the software, find bugs and create a ticket? Can an AI agent use a browser, try to use the software and suggest new features?
I’ve come to the realization that these kind of systems don’t work, and that a human in the loop is crucial for task planning; the LLM’s role being to identify issues, communicate the design / architecture, etc before it’s handed off, otherwise the LLM always ends up doing not entirely the correct thing. How is this part tackled when all that you have is GH issues? Doesn’t this work only for the most trivial issues?
I’ve come to the realization that these kind of systems don’t work, and that a human in the loop is crucial for task planning; the LLM’s role being to identify issues, communicate the design / architecture, etc before it’s handed off, otherwise the LLM always ends up doing not entirely the correct thing. How is this part tackled when all that you have is GH issues? Doesn’t this work only for the most trivial issues?
Had the same realization which inspired eforge (shameless plug) https://github.com/eforge-build/eforge - planning stays in the developer’s control with all engineering (agent orchestration) handed off to eforge. This has been working well for a solo or siloed developer (me) that is free to plan independently. Allows the developer to confidently stay in the planning plane while eforge handles the rest using a methodol…
Earlier quoted context omitted.
Had the same realization which inspired eforge (shameless plug) https://github.com/eforge-build/eforge - planning stays in the developer’s control with all engineering (agent orchestration) handed off to eforge. This has been working well for a solo or siloed developer (me) that is free to plan independently. Allows the developer to confidently stay in the planning plane while eforge handles the rest using a methodol…
To me that doesn't do enough yet in terms of up-front planning and visualization, but it's a step in the right direction. I prefer Traycer myself.