Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Agents need control flow, not more prompts

bsuh.bearblog.dev

Post reply on HN
Print view View original

271–280 of 348 posts

Re: Agents need control flow, not more prompts

#271

Post by LikesPwsh »

krzyk wrote: post #258
fny wrote: post #229

Earlier quoted context omitted.

Secret: "compile" that orchestration prompt. Determinism is solved by turning prompts into code that can in turn run agents or run code or both. Everyone misses this pattern with skills: you can just drop code alongside a SKILL.md to guarantee certain behaviors, but for some reason everyone's addicted to writing prompts. You don't even need to build a CLI. A simple skill.py with tasks does it. You can even have helpe…

Could you elaborate what does "compiling orchestration prompt" mean?

Rather than telling the LLM "loop through these files", tell it "write a script to loop through these files", then hard-code that script somewhere.

Re: Agents need control flow, not more prompts

#272

Post by beshrkayali »

Humble mention, I’ve been thinking the same thing with Ossature for the last couple of months since I started working on it: https://ossature.dev

The models are already good enough for code generation. What we need is the harness around them actually deterministically enforcing a specific path and “leashing” the models output to be aligned with the intention of the user as much as possible. You can’t make the output of the model deterministic, but you can make everything around it to be so.

Trying to make enforcements work with prompts is like a government agency investigating/auditing itself, there’s no incentive to find problems, so you’ll always inevitably get the “All Good, Boss!”

Re: Agents need control flow, not more prompts

#273

Post by toasty228 »

beshrkayali wrote: post #272

Humble mention, I’ve been thinking the same thing with Ossature for the last couple of months since I started working on it: https://ossature.dev The models are already good enough for code generation. What we need is the harness around them actually deterministically enforcing a specific path and “leashing” the models output to be aligned with the intention of the user as much as possible. You can’t make the output…

> so you’ll always inevitably get the “All Good, Boss!”

Or the opposite depending on how you ask the questions, some automated code review tools _always_ find issues, even when they don't really exist, or they exist in the scope of a function but not once the function is wired in the project.

Re: Agents need control flow, not more prompts

#274

Post by HatchedLake721 »

memjay wrote: post #233
rnxrx wrote: post #22

I wonder if a part of the problem isn't just the misapplication of LLMs in the first place. As has been mentioned elsewhere, perhaps the agent's prompt should be to write code to accomplish as much of the task in as repeatable/verifiable/deterministic a way as possible. This would hopefully include validation of the agent's output as well. The overall goal would be to keep the LLM out of doing processing that could b…

This has been our experience as well. Initially we had a list of tools that the agent could use to manipulate a data structure in certain ways. This approach was quite brittle. Now we are using a small DSL (domain specific language) and a single tool where the agent can input scripts written in the DSL. We are getting more dynamic use-cases now and wrong syntax can easily be catched by the parser and relayed to the a…

Do you have an example of type of data and DSL? I feel I’d just give it access to write python/js to manipulate data

Re: Agents need control flow, not more prompts

#277

Post by dkersten »

user34283 wrote: post #265
dkersten wrote: post #251

This is something I realised late last year while using Claude Code. The LLM shouldn't be the one in control of the workflow, because the LLM can make mistakes, skip steps, hallucinate steps, etc. Its also wasteful of tokens. I'm a firm believer that a "thin harness" is the wrong approach for this reason and that workflows should be enforced in code. Doing that allows you to make sure that the workflow is always foll…

I‘ve recently started to use skills and so far it’s been working great. Your agent can write a python script to loop and simply call „claude -p“ or „codex exec“. For simple workflows this seems good enough and can be set up in 10 minutes without third party software. What do you think?

For simple workflows or once-off workflows, that's a good approach.

For long running repeatable workflows (eg you want to leave your agent running over night, you want to run the same workflows over and over in different projects, or more autonomous Devin-like workflows) or you want audit trails/observability, vetted workflows (ie not have the LLM write them; or have the LLM write them and you review them) without having to read through scripts, or you have more complex requirements like having different models/providers for different workflow stages or the things I mentioned previously (context, plans, verification, etc), or you have more complex workflow needs (swarms or fork/join, parallel pipelines, routing/branching, error recovery or routing, etc) then a robust dedicated workflow engine is needed in my personal opinion.

I think for most users using claude/codex for themselves on smallish projects, its unnecessary, but was you scale up, I feel that more powerful tools are needed. Also, for corporate, where you need repeatable workflows with audit trails, artefact management, and job queue based task management starts becoming more important too.

I also feel that using a workflow engine as an internal behind-the-scenes system in a GUI-centric vibe coding tool might also help raise the ceiling compared to the existing tools, but I've yet to test that hypothesis. Just because it takes the mistakes out of the users hands: the engine will follow proven workflows, whether you ask it to or not, keeping skills for context/knowledge, not for orchestration.

Something else I've been experimenting with a little, but not enough yet to have an opinion, is small language models running locally for orchestration, and frontier models for doing work.

Re: Agents need control flow, not more prompts

#278

Post by quinnjh »

reddit_clone wrote: post #144
sisve wrote: post #87

Earlier quoted context omitted.

Yeah, the problem is that I do not think the agents is good at reusing scripts and stitching it together.At least for me it's recreating to much similar. I hope we will see platforms like windmill.dev find the optimal solution for this. I have not been able to test it enough. But have a platform that gives you some observability out of the box and protect secrets from llm is nice

I noticed that too. Unless you _ask_ for a script, they throw away the scripts they write. They are particularly bad at complex multiline parsing. Writing all sorts of weird/crude python/awk scripts and getting confused in the process. I wish they would use Perl6/Grammer or Haskell/Parsec or similar and write better parsing scripts.

For the non haskell folks like myself, what would that look like/ why is parsing better? Perl i get

Re: Agents need control flow, not more prompts

#279

Post by robinduckett »

fny wrote: post #229
827a wrote: post #100

1000% agree. I am increasingly hesitant to believe Anthropic's continual war drum of "build for the capabilities of future models, they'll get better". We've got a QA agent that needs to run through, say, 200 markdown files of requirements in a browser session. Its a cool system that has really helped improve our team's efficiency. For the longest time we tried everything to get a prompt like the following working: "…

Secret: "compile" that orchestration prompt. Determinism is solved by turning prompts into code that can in turn run agents or run code or both. Everyone misses this pattern with skills: you can just drop code alongside a SKILL.md to guarantee certain behaviors, but for some reason everyone's addicted to writing prompts. You don't even need to build a CLI. A simple skill.py with tasks does it. You can even have helpe…

Exactly this, I tend to work this way. I built an ingestion pipeline to pull concepts out of a novel using Qwen and push them into falkordb this way

Re: Agents need control flow, not more prompts

#280

Post by vishna »

827a wrote: post #100

1000% agree. I am increasingly hesitant to believe Anthropic's continual war drum of "build for the capabilities of future models, they'll get better". We've got a QA agent that needs to run through, say, 200 markdown files of requirements in a browser session. Its a cool system that has really helped improve our team's efficiency. For the longest time we tried everything to get a prompt like the following working: "…

This was a great example, thanks
Post reply on HN