Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Optimizing Tool Selection for LLM Workflows with Differentiable Programming

viksit.substack.com

Post reply on HN
Print view View original

31–40 of 43 posts

Re: Optimizing Tool Selection for LLM Workflows with Differentiable Programming

#31

Post by viksit »

lgas wrote: post #28
nphard85 wrote: post #23

Very interesting. How does this approach work for complex agentic workflows where the LLM is expected to orchestrate across multiple tools (such as when using MCP)? Or is this mainly for simple cases like the ones presented in the blog post?

The work described appears as if it would handle a complex set of multiple tools just fine, but you do train the controller on a specific tool set, so you would presumably need to train (or at least something like "fine tune") a controller for each toolset you wanted to use.

for sure, there's a way here where I think we ought to be able to learn multiple tool calls and prompts together with real world data. investigating that next.

Re: Optimizing Tool Selection for LLM Workflows with Differentiable Programming

#33

Post by crazylogger »

I can see this makes sense for simple { user_query -> search -> llm_answer } usage, where tool use is only a means to retrieve background info.

For complex real-world agent flows though, tool use is often the only thing that the LLM is expected to do. Like in a coding agent:

```

User: Develop a program to ...

Agent: Bash("touch main.py") > 0, ""

Agent: Edit("main.py", initial_patch) > 0, ""

Agent: Bash("python main.py") > 1, "SyntaxError: ..."

Agent: Edit("main.py", fix_patch) > 0, ""

Agent: Bash("python main.py") > 0, "OK"

Agent: FINISH

```

Here, tool selection (+ writing the arguments) is actually the whole job. It's also easy to see that if you omit even one of the tool use records in the middle, the agent wouldn't work at all.

Re: Optimizing Tool Selection for LLM Workflows with Differentiable Programming

#34

Post by pcwelder »

You've essentially just trained your own LM instead of using a pretrained large LM.

Speaking generically -- any place in your workflow you feel the task is not hard, you can use smaller and cheaper LM.

Smaller LMs come with accuracy reduction, particularly in tail cases. So in the real world this doesn't work out.

Also is gumble softmax usage intentional? It looks like a straightforward classifier that just needs regular softmax.

Re: Optimizing Tool Selection for LLM Workflows with Differentiable Programming

#36

Post by rybosome »

viksit wrote: post #17
rybosome wrote: post #13

Earlier quoted context omitted.

Thanks for the informative and inspiring post! This is definitely cool, and I can imagine very useful. However I do want to mention that the “recommended” flow these days isn’t to separate out a tool request in the way you have. Eg instead of asking an LLM to route a tool, extracting that, running the tool, passing output back to the LLM, etc. - you simply pass the tool definitions, prompt, structural output expectat…

great point, appreciate the comment. totally agree with your framing, though i think there’s still a gap in how tool use is handled today. quick note: it doesn’t have to be an rnn. i’ve got a follow-up example coming that uses a transformer-style ToolController with self attention, more expressive routing, etc. but here’s the thing — when you rely on few-shot bootstrapping the LLM, you never end up updating the model…

That’s cool, I’d love to see the advanced ToolController when it’s available!

Great points about not updating priors. I also thought about it a bit more and realized that there’s a way you can largely mitigate the out-of-distribution inference requests after local tool selection, if you wanted to.

The tool use loop in an inference framework builds up history of each interaction and sends that along with each subsequent request. You could create “synthetic history”, where you send the LLM history containing the prompt, your local tool selection masquerading as though the LLM generated it, and the tool response. This would be in-distribution but still rely on your local tool routing.

If this works well enough, then I think your approach is very powerful once you’ve decided on a task and set of tools and are able to commit to training on that. Definitely want to try this myself now.

Looking forward to seeing more! I take it your substack is the best place to follow along?

Re: Optimizing Tool Selection for LLM Workflows with Differentiable Programming

#37

Post by bigmadshoe »

This is super cool!

From the article:

  Each LLM call incurs latency, cost, and token overhead. More subtly, it compounds context:
  every step includes not only the original query, but intermediate outputs and scratchpad logic from earlier prompts. 
  This creates a growing burden on both inference and model performance.
I was working with agents over a year ago before the common workflows had really been set in stone. At that time we were heavily doctoring the context to give a very streamlined representation of what had occurred during a given run to the LLM. Is this not standard practice?

Re: Optimizing Tool Selection for LLM Workflows with Differentiable Programming

#38

Post by viksit »

rybosome wrote: post #36
viksit wrote: post #17

Earlier quoted context omitted.

great point, appreciate the comment. totally agree with your framing, though i think there’s still a gap in how tool use is handled today. quick note: it doesn’t have to be an rnn. i’ve got a follow-up example coming that uses a transformer-style ToolController with self attention, more expressive routing, etc. but here’s the thing — when you rely on few-shot bootstrapping the LLM, you never end up updating the model…

That’s cool, I’d love to see the advanced ToolController when it’s available! Great points about not updating priors. I also thought about it a bit more and realized that there’s a way you can largely mitigate the out-of-distribution inference requests after local tool selection, if you wanted to. The tool use loop in an inference framework builds up history of each interaction and sends that along with each subseque…

this is really interesting! yes, its my substack.

also, if you're down, love to connect and talk more about what use cases / techniques you're using. I'm @viksit on X dms if that works.

Re: Optimizing Tool Selection for LLM Workflows with Differentiable Programming

#39

Post by viksit »

joe_the_user wrote: post #4
viksit wrote: post #2

I was experimenting with how local, learnable routers can reduce token overhead, and lower costs, and decided to publish a post about it. The main goal is to delegate tool calls via a PyTorch based learner and examples of how to integrate this into a DSPy pipeline. Feedback welcome!

My question is whether you have managed to make this work, perform a specific complex task, in some real world situation.

great q. thats coming up as a future post in the series.

Re: Optimizing Tool Selection for LLM Workflows with Differentiable Programming

#40

Post by viksit »

bigmadshoe wrote: post #37

This is super cool! From the article: Each LLM call incurs latency, cost, and token overhead. More subtly, it compounds context: every step includes not only the original query, but intermediate outputs and scratchpad logic from earlier prompts. This creates a growing burden on both inference and model performance. I was working with agents over a year ago before the common workflows had really been set in stone. At…

yes, AFAIK right now, there are no easy ways of "slimming" context because no one knows what it should be or how.
Post reply on HN