LLM can write programs in any programming language it knows about. So how about askinng it to write a shell-program that does the tool-calls on the client? You might want to run in some kind of sandbox to prevent the LLM from taking over the world, security is an isssue. But apart from that why not make the LLM write shell-programs instead of relying on JSON etc. ? Shell-scripting is the language for controlling the…
Not always shell—Python, often a subset, is common—but a single sandboxed coding-running tool as the way to run all of the other things that would otherwise be their own top-level tools themselves seems an increasingly common approach.
Once permissions are safe and secure and easy to set up and understand then using shell-programming as a general interface between agents and your PC/OS might be a good option.
Come to think of it I recently read that Microsoft is planning to produce an AI-oriented OS. Maybe agents with user-accounts is something they are aiming for, to solve the problem discussed in the article.