Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The M×N problem of tool calling and open-source models

thetypicalset.com

Post reply on HN
Print view View original

11–20 of 56 posts

Re: The M×N problem of tool calling and open-source models

#11

Post by yorwba »

perlgeek wrote: post #9
akoumjian wrote: post #8

Earlier quoted context omitted.

The models only output text. Tool calls are nothing more than specially formatted text which gets parsed and interpreted by the inference server (or some other driver) into something which can be picked up by your agent loop and executed. Models are trained in a wide variety of different delimiters and escape characters to indicate their tool calls (along with things like separate thinking blocks). MCP is mostly a st…

> Tool calls are nothing more than specially formatted text which gets parsed and interpreted by the inference server I know this is getting off-topic, but is anybody working on more direct tool calling? LLMs are based on neural networks, so one could create an interface where activating certain neurons triggers tool calls, with other neurons encoding the inputs; another set of neurons could be triggered by the token…

Each text token already represents the activation of certain neurons. There is nothing "more direct." And you cannot fully separate data and metadata if you want them to influence the output. At best you can clearly distinguish them and hope that this is enough for the model to learn to treat them differently.

Re: The M×N problem of tool calling and open-source models

#12

Post by HarHarVeryFunny »

evelant wrote: post #7

I guess I fail to see why this is such a problem. Yes it would be nice if the wire format were standardized or had a standard schema description, but is writing a parser that handles several formats actually a difficult problem? Modern models could probably whip up a "libToolCallParser" with bindings for all popular languages in an afternoon. Could probably also have an automated workflow for adding any new ones with…

There already exist products like LiteLLM that adapt tool calling to different providers. FWIW, incompatibility isn't just an opensource problem - OpenAI and Anthropic also use different syntax for tool registration and invocation.

I would guess that lack of standardization of what tools are provided by different agents is as much of a problem as the differences in syntax, since the ideal case would be for a model to be trained end-to-end for use with a specific agent and set of tools, as I believe Anthropic do. Any agent interacting with a model that wasn't specifically trained to work with that agent/toolset is going to be at a disadvantage.

Re: The M×N problem of tool calling and open-source models

#17

Post by jeremyjh »

HarHarVeryFunny wrote: post #12
evelant wrote: post #7

I guess I fail to see why this is such a problem. Yes it would be nice if the wire format were standardized or had a standard schema description, but is writing a parser that handles several formats actually a difficult problem? Modern models could probably whip up a "libToolCallParser" with bindings for all popular languages in an afternoon. Could probably also have an automated workflow for adding any new ones with…

There already exist products like LiteLLM that adapt tool calling to different providers. FWIW, incompatibility isn't just an opensource problem - OpenAI and Anthropic also use different syntax for tool registration and invocation. I would guess that lack of standardization of what tools are provided by different agents is as much of a problem as the differences in syntax, since the ideal case would be for a model to…

Presumably the hosting services are resolving all of this in their OpenAI/Anthropic compatibility layer, which is what most tools are using. So this is really just a problem for local engines that have to do the same thing but are expected to work right away for every new model drop.

Re: The M×N problem of tool calling and open-source models

#18

Post by perlgeek »

yorwba wrote: post #11
perlgeek wrote: post #9

Earlier quoted context omitted.

> Tool calls are nothing more than specially formatted text which gets parsed and interpreted by the inference server I know this is getting off-topic, but is anybody working on more direct tool calling? LLMs are based on neural networks, so one could create an interface where activating certain neurons triggers tool calls, with other neurons encoding the inputs; another set of neurons could be triggered by the token…

Each text token already represents the activation of certain neurons. There is nothing "more direct." And you cannot fully separate data and metadata if you want them to influence the output. At best you can clearly distinguish them and hope that this is enough for the model to learn to treat them differently.

Are there tokens reserved for tool calls? If yes, I can see the equivalence. If not, not so much.

Re: The M×N problem of tool calling and open-source models

#20

Post by Leon8090 »

evelant wrote: post #7

I guess I fail to see why this is such a problem. Yes it would be nice if the wire format were standardized or had a standard schema description, but is writing a parser that handles several formats actually a difficult problem? Modern models could probably whip up a "libToolCallParser" with bindings for all popular languages in an afternoon. Could probably also have an automated workflow for adding any new ones with…

[dead]
Post reply on HN