Isn’t this basically what Palantir does?
Agents need control flow, not more prompts
201–210 of 348 posts
Re: Agents need control flow, not more prompts
#202Earlier quoted context omitted.
> can you deterministically test the thing you are asking it to do? Of course: have it write tests first; and run them to check its work. Works well for refactoring, but greenfield implementations still rely on a spec that is guaranteed to be incomplete, overcomplete and wrong in many ways.
You can't ask something to check its own work without external reward/penalty. It'll cheat.
Weirdly, and i fully think this is just some cognitive bias I don't have the knowledge to name, the ai seems very happy to please me. Like when it gets something done in one shot, it seems very happy to do so.
Re: Agents need control flow, not more prompts
#203I wonder if a part of the problem isn't just the misapplication of LLMs in the first place. As has been mentioned elsewhere, perhaps the agent's prompt should be to write code to accomplish as much of the task in as repeatable/verifiable/deterministic a way as possible. This would hopefully include validation of the agent's output as well. The overall goal would be to keep the LLM out of doing processing that could b…
We have a rule that the LLM cannot perform any actions that result in actual money or stuff moving. Those can only be done by API calls that have lots of validation and checks on them, and adding or changing an API call is gated behind human review. The LLM is then free to make as many API calls as it likes, we're confident that it can't screw anything up too badly.
Re: Agents need control flow, not more prompts
#204> Babysitter: Keep a human in the loop to catch errors before they propagate. This is the only way to guarantee AI usage doesn't burn you. Any automation beyond this is just theater, no matter how much that hurts to hear/undermines your business model. A bird sings, a duck quacks. You don't expect the duck to start singing now, do you?
I think babysitting LLMs is exactly the thing that burns you.
Presuming you meant burns you out though.
Re: Agents need control flow, not more prompts
#2051000% agree. I am increasingly hesitant to believe Anthropic's continual war drum of "build for the capabilities of future models, they'll get better". We've got a QA agent that needs to run through, say, 200 markdown files of requirements in a browser session. Its a cool system that has really helped improve our team's efficiency. For the longest time we tried everything to get a prompt like the following working: "…
Wouldn't it be more efficient to convert the requirements these 200 markdown files into Playwright tests?
You could still use an LLM to write and extend the tests, but running the tests would be deterministic and would use less resources.
Re: Agents need control flow, not more prompts
#206deepwork.md is made for this.
Re: Agents need control flow, not more prompts
#207I'm building a robust runtime for this.
It's externally orchestrated and managed, not by an agent running the the loop.
The goal is to force LLMs to produce exactly what you want every time.
I will be open sourcing soon. You can use whatever harness or tools you already use, you just delegate the actual implementation to the engine.
Re: Agents need control flow, not more prompts
#2081000% agree. I am increasingly hesitant to believe Anthropic's continual war drum of "build for the capabilities of future models, they'll get better". We've got a QA agent that needs to run through, say, 200 markdown files of requirements in a browser session. Its a cool system that has really helped improve our team's efficiency. For the longest time we tried everything to get a prompt like the following working: "…
Wouldn't it be more efficient to convert the requirements these 200 markdown files into Playwright tests? You could still use an LLM to write and extend the tests, but running the tests would be deterministic and would use less resources.
This type of thing so much.
AI is being pushed so much at work right now. For non-dev stuff even. The amount of things that people think are "awesome never seen this" is staggering.
Just because you haven't seen file format X converted to file format Y before and now you asked the LLM to do it and it worked, doesn't mean you needed an LLM for it nor that it's remarkable. The LLM knew how to do it because it learned from a bazillion online sources for deterministic converters that cost nothing (and have open source). But now you're paying, every single time, for a non-deterministic version of it and you find it cool. It's magic ...
But I guess they deserve it.
Re: Agents need control flow, not more prompts
#209Build CLIs your agents call, that scaffold what you want, and lint so it actually does achieves your intended design. Markdown files are a good reference but they are a weak enforcement tool and go stale easily. Avoid burying yourself in more skills docs you’re not even writing yourself and probably never even read. Focus that toward deterministic tooling. (Not that skills or prompts are bad, I agree a meta skill tha…
lol so write an actual deterministic program? we're close to full circle
Yes but with the "judgement" to call them. If you put "review the results based on conditions described here and anything else suspicious you may spot before call the ", it should be able to catch some case you didn't think about in your standard checks. Of course it may miss out on those or have false positives but that is the nature of the beast, as it is now.
Re: Agents need control flow, not more prompts
#2109 different frameworks being pushed in the comments of this thread. 2026 truly is the year of agents.