The reliance on context to drive correct actions just doesn't work well. I am constantly wrestling with AI agents that do not do what you tell them. Every AI agent out there seems to suck in this regard, leaving it up to the user to build in their own guardrails. I have a bad feeling that nobody is working on an improved solution.
I’ve seen no reason to believe it’s even possible to solve this. The worst thing about LLMs is they can pass the Turing test, leading people to believe they have an Asimov style robot instead of a very cool statistical model. It feels like they should be able to follow instructions or keep instructions from content separate, but that’s not what’s happening.
So really, your tool-specific rules should be passed to the AI either with your follow-up prompt, or in response to the request to issue a tool call, so the AI can validate what it will compose the tool call as, right as it's making the call. This means the agent should keep track of tool-specific rules, and reinforce them to the AI. Yes this will spend a few more tokens per call, but it will probably improve the outcomes somewhat.
In addition to this, we should probably be abstracting the tool calls more. Rather than let the AI run a Bash one-liner which includes writing files to `/tmp/foo.txt`, we should have the AI output even more structured tool calls, liike `make_temp_file AS BAR`, and have it then call another tool referencing $BAR (`some_other_tool -tmpfile $BAR`). This way there is less to go wrong because it's not getting in the weeds doing shell scripting while it's trying to do something more important (diagnosing an issue).
I think this will require additional training by the AI companies. Which is why we need to define these kind of standards now, so 6-12 months from now, we will have AI that actually support these higher level abstractions. You then customize your abstraction, and the AI doesn't have to know anything about how it works on your box. It would greatly reduce the complexity required for AIs to do agentic work.