Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Agents need control flow, not more prompts

bsuh.bearblog.dev

Post reply on HN
Print view View original

331–340 of 348 posts

Re: Agents need control flow, not more prompts

#331

Post by bostonvaulter2 »

bob1029 wrote: post #184
827a wrote: post #100

1000% agree. I am increasingly hesitant to believe Anthropic's continual war drum of "build for the capabilities of future models, they'll get better". We've got a QA agent that needs to run through, say, 200 markdown files of requirements in a browser session. Its a cool system that has really helped improve our team's efficiency. For the longest time we tried everything to get a prompt like the following working: "…

I saw a major uplift in performance after I combined tools like apply_patch with check_compilation & run_unit_tests. I still call the tool "apply_patch", but it now returns additional information about the build & tests if the patch succeeds. The agent went from ~80% success rate to what seems to be deterministic (so far). I don't bother to describe the compilation and unit testing processes in my prompts anymore. Al…

Can you explain in more detail how you implemented those tools? Is that via a MCP server?

Re: Agents need control flow, not more prompts

#333

Post by memjay »

HatchedLake721 wrote: post #274
memjay wrote: post #233

Earlier quoted context omitted.

This has been our experience as well. Initially we had a list of tools that the agent could use to manipulate a data structure in certain ways. This approach was quite brittle. Now we are using a small DSL (domain specific language) and a single tool where the agent can input scripts written in the DSL. We are getting more dynamic use-cases now and wrong syntax can easily be catched by the parser and relayed to the a…

Do you have an example of type of data and DSL? I feel I’d just give it access to write python/js to manipulate data

We decided not to go with Python/JS to make executing safe and simple.

The data structure is a recursive list of simple objects that form a table of content.

DSL uses Python syntax though. For example:

swap_section(a, b) create_section(after=2) delete_section(2)

This proved to safe a ton of explanatory prompts that would be needed if every command was a tool instead. And it’s faster and more reliable.

Re: Agents need control flow, not more prompts

#334

Post by cadamsdotcom »

Yep. Deterministic shell around the powerful abilities of the model.

Define what a good job looks like, unskippable steps, etc etc - essentially what your process is for producing your desired output in a reproducible way.

Then codify it. Have the model write code and wrap the model in a harness that ensures said code runs when you need it to, every time.

Re: Agents need control flow, not more prompts

#335

Post by sidcool »

philipp-gayret wrote: post #248
sidcool wrote: post #247

How does one achieve this?

Native integrations with agents, i.e. Claude Code's system of Hooks. Harnesses, which kick off agents with what to do. Tools, which show an agent where in a process it is, and what the next step should be. In my experience I find Hooks to be extremely powerful cross-project. CLI Tools are easy to make also, and work really well for guiding agents.

Thanks. Any tutorials?

Re: Agents need control flow, not more prompts

#337

Post by philipp-gayret »

sidcool wrote: post #335
philipp-gayret wrote: post #248

Earlier quoted context omitted.

Native integrations with agents, i.e. Claude Code's system of Hooks. Harnesses, which kick off agents with what to do. Tools, which show an agent where in a process it is, and what the next step should be. In my experience I find Hooks to be extremely powerful cross-project. CLI Tools are easy to make also, and work really well for guiding agents.

Thanks. Any tutorials?

For Hooks (IMO the most powerful feature) Id recommend only https://code.claude.com/docs/en/hooks-guide and for Plugins, Skills, MCP and so on the official documentation by Anthropic has been the source you'd need. As for harnesses and CLI tools Id go with whatever you're already familiar with, can't make a particular recommendation.

Re: Agents need control flow, not more prompts

#338

Post by suttontom »

whattheheckheck wrote: post #296
LikesPwsh wrote: post #271

Earlier quoted context omitted.

Rather than telling the LLM "loop through these files", tell it "write a script to loop through these files", then hard-code that script somewhere.

The models will eventually be able to know that they need to do that to get the thing done from natural language

"The models will eventually..." Yeah but they haven't, and it's been years now. Also who cares? We have problems right now that need to be solved.

Re: Agents need control flow, not more prompts

#339

Post by bob1029 »

bostonvaulter2 wrote: post #331
bob1029 wrote: post #184

Earlier quoted context omitted.

I saw a major uplift in performance after I combined tools like apply_patch with check_compilation & run_unit_tests. I still call the tool "apply_patch", but it now returns additional information about the build & tests if the patch succeeds. The agent went from ~80% success rate to what seems to be deterministic (so far). I don't bother to describe the compilation and unit testing processes in my prompts anymore. Al…

Can you explain in more detail how you implemented those tools? Is that via a MCP server?

> Is that via a MCP server?

No, this all in one application. A Winforms+WebView2 app wraps the chat completion APIs and implements the various tools directly.

Re: Agents need control flow, not more prompts

#340

Post by venturin »

Strong agreement on the thesis. The piece is most useful for naming what's bankrupt about prompt chains. Where it stops short is what the verification checkpoints should actually verify.

One way to slice it: there are three kinds of underspecification an agent has to close.

Intent: what the user wanted (JWT vs cookies, should free users see this feature). Verification can't close this and probably shouldn't try.

Structural e.g. null, types, exhaustiveness, ownership. Sound static analysis closes this by construction.

Domain e.g. auth on every route, error propagation, contract stability. A domain-shaped apparatus closes this because it knows what kind of program is being built.

Babysitter, auditor, prayer is the right taxonomy of bad options. The fourth option is making the LLM a component inside an apparatus that handles structural and domain statically, and leaving the human on intent.

Post reply on HN