Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Optimizing Tool Selection for LLM Workflows with Differentiable Programming

viksit.substack.com

Post reply on HN
Print view View original

21–30 of 43 posts

Re: Optimizing Tool Selection for LLM Workflows with Differentiable Programming

#21

Post by viksit »

Garlef wrote: post #5

Is selection really the issue? You'd still need to figure out what payload to give to the tool based on your context. But I guess depending on your business case it might be worth it. It's not something I'd do from the beginning, though.

it’s not just about selection. say you’ve got 100k tool calls — in the current hosted llm setup, you don’t actually learn anything new about your data to improve future tool accuracy.

this gets worse when you’re chaining 3–4+ tools. context gets noisy, priors stay frozen and there's prompt soup..

my intuition here is: you can learn the tool routing and the llm prompts before and after the call. (can always swap out the rnn for a more expressive encoder model and backprop through the whole thing).

super useful when you’re building complex workflows -- it gives you a way to learn the full pipeline, not just guess and hope.

Re: Optimizing Tool Selection for LLM Workflows with Differentiable Programming

#22

Post by viksit »

phanimahesh wrote: post #16
Garlef wrote: post #5

Is selection really the issue? You'd still need to figure out what payload to give to the tool based on your context. But I guess depending on your business case it might be worth it. It's not something I'd do from the beginning, though.

This is a bigger problem than it looks like at first glance. For isecases where llm + tool calls make more sense compared to say llm assisted codegen, figuring out the tool arguments is nontrivial. Where it is relatively easy I think codegen is a better option wrt amortised running costs

this is a great point, ty.

in my mind the biggest difference is llms that are invoked during a workflow, and llms that are invoked when _creating_ code (codegen).

for the former, tools could be well defined till they are small in number, but at some point, the system needs to examine a library of tools, understand how to call it and integrate it, and at its peak, even create new tools to talk to systems not already present in that library (codegen).

Re: Optimizing Tool Selection for LLM Workflows with Differentiable Programming

#24

Post by viksit »

bGl2YW5j wrote: post #12

I don’t think the problem is “how to optimise tool selection for the LLM”. I think the real problem is using an LLM to do tool selection at all. This is control flow and I believe should be handled with hardcoded rules and/separation of concerns. If LLMs could handle determinism better, I’d say having a single chat-based entrypoint into a plethora of services makes sense. But as they stand, it doesn’t make sense. Sim…

+1 on the control flow point.

I think of an llm as a differentiable interpreter of a program. it should do decision making (tool selection, argument routing), branching logic via weights + gates etc.

so as a differentiable state machine:

- each state == a stage in your workflow

- transitions == tool calls

- encode this as a rnn or graph

and learn transitions and actions via supervision or RL

Re: Optimizing Tool Selection for LLM Workflows with Differentiable Programming

#25

Post by viksit »

apsears wrote: post #14

I have been thinking a lot about tool selection lately, and something that I keep repeating to myself is: "the LLM has intuition, but I have data". I guess that applies when you're not able to fine-tune the LLM you're using. Presumably Anthropic has a lot of data too.

+1 - the biggest issue is not being able to fine tune the llm to learn the specifics of how to make a tool call better over time, which an approach like this can bring to the table.

Re: Optimizing Tool Selection for LLM Workflows with Differentiable Programming

#27

Post by viksit »

tomlue wrote: post #6

you could also propagate loss into the tools themselves.

+1 - you can propagate the loss for a workflow across prompts + tools, which would make it much better to do resilient workflows. or "agents" as everyone calls them now ;)

Re: Optimizing Tool Selection for LLM Workflows with Differentiable Programming

#28

Post by lgas »

nphard85 wrote: post #23

Very interesting. How does this approach work for complex agentic workflows where the LLM is expected to orchestrate across multiple tools (such as when using MCP)? Or is this mainly for simple cases like the ones presented in the blog post?

The work described appears as if it would handle a complex set of multiple tools just fine, but you do train the controller on a specific tool set, so you would presumably need to train (or at least something like "fine tune") a controller for each toolset you wanted to use.

Re: Optimizing Tool Selection for LLM Workflows with Differentiable Programming

#29

Post by viksit »

nphard85 wrote: post #23

Very interesting. How does this approach work for complex agentic workflows where the LLM is expected to orchestrate across multiple tools (such as when using MCP)? Or is this mainly for simple cases like the ones presented in the blog post?

+1 thanks for mentioning MCP!

re: different tools (apis vs mcps). in my mind, there should be no real difference at what kind of tools is called at this moment since I model this as a softmax over a label set of tools.

that said, an idea I want to investigate is whether tools can live in a learned embedding space, where selection isn’t a softmax over discrete labels but a nearest-neighbor or attention mechanism over continuous vectors.

this is the intuition I'm developing as we speak and in some of my other comments on this thread (see differentiable state machine comment).

Post reply on HN