I was afraid my description sounded like agents, and it kind of is, but not like most implementations. Most use a “boss bot” to craft a prompt/system message and launch the model, and sometimes they redo it every time it launches the agent. It’s a low effort attempt that’s immediately flawed because it uses LLM output for LLM input. It can look like it’s working for some time, but the perplexity guarantees it’s a roll of the dice. That’s what eats away at these kinds of projects. “It was doing great until it rm’d prod.”
Run the same exact prompt 100x in a single step test (one prompt, one response), hash the full responses, and you’ll see 10-50+ unique responses. The higher the unique the worse your prompt; focus on the system. A highly tuned system prompt will result in one response, even at a temperature of 1.0. Really. Once that’s done that’s the only thing it does and it’s the only one that does it and it never changes. Other LLMs that call ‘check_email()’ are unknowingly just passing a prompt to the specialized one.
I use the API directly, craft a small Python script for the API call and task interface, then hyper-optimize the system/user prompt using test scenarios and automated loops. My system prompts rarely/never contain complete sentences, yet include all the tools/functions and requirements.
Make your error messages user prompt instructions, not errors. That’s why “agent optimized” models exist. Chat models are primarily trained on conversational text, meaning the stackoverflow “How do I fix ‘too many levels of symbolic links’?” -> Explanation/resolution. It’s far less on “# ls broken_loop” -> “# ls: cannot access ‘broken_loop’: Too many levels of symbolic links” -> “# namei -l broken_loop”
It’s not that the good ones are bad, but you’re leaning on the million training documents rather than the trillion.
Anyway, go that route with your system. Think about it more like automating a factory floor rather than hiring interns.
The least efficient methods, by definition, have the most room for improvement, which means they have the greatest reward potential, but for that one “eureka” moment. The path less taken is often interesting, but the ill-advised path still has fruit on the trees.