Optimizing Tool Selection for LLM Workflows with Differentiable Programming
viksit.substack.com
Optimizing Tool Selection for LLM Workflows with Differentiable Programming
1–10 of 43 posts
Re: Optimizing Tool Selection for LLM Workflows with Differentiable Programming
#2I was experimenting with how local, learnable routers can reduce token overhead, and lower costs, and decided to publish a post about it. The main goal is to delegate tool calls via a PyTorch based learner and examples of how to integrate this into a DSPy pipeline. Feedback welcome!
Re: Optimizing Tool Selection for LLM Workflows with Differentiable Programming
#3I was experimenting with how local, learnable routers can reduce token overhead, and lower costs, and decided to publish a post about it. The main goal is to delegate tool calls via a PyTorch based learner and examples of how to integrate this into a DSPy pipeline. Feedback welcome!
I think this is a creative approach. I wonder how the success rates for that little RNN compare to the success rates of the primary LLM, especially for complex queries or complex tool calls. At some point you have to scale that network up large enough to get better results. Eventually you've come back around and you might as well use an LLM. I think a similar approach with potentially better results (depends on the application) could be accomplished by using that same dataset to finetune a small language model. It'd be interesting to see some success rate comparisons.
Re: Optimizing Tool Selection for LLM Workflows with Differentiable Programming
#4I was experimenting with how local, learnable routers can reduce token overhead, and lower costs, and decided to publish a post about it. The main goal is to delegate tool calls via a PyTorch based learner and examples of how to integrate this into a DSPy pipeline. Feedback welcome!
My question is whether you have managed to make this work, perform a specific complex task, in some real world situation.
Re: Optimizing Tool Selection for LLM Workflows with Differentiable Programming
#5Is selection really the issue?
You'd still need to figure out what payload to give to the tool based on your context.
But I guess depending on your business case it might be worth it. It's not something I'd do from the beginning, though.
Re: Optimizing Tool Selection for LLM Workflows with Differentiable Programming
#6you could also propagate loss into the tools themselves.
Re: Optimizing Tool Selection for LLM Workflows with Differentiable Programming
#7I was experimenting with how local, learnable routers can reduce token overhead, and lower costs, and decided to publish a post about it. The main goal is to delegate tool calls via a PyTorch based learner and examples of how to integrate this into a DSPy pipeline. Feedback welcome!
Nit - code screenshots are a PITA to read on mobile!
Re: Optimizing Tool Selection for LLM Workflows with Differentiable Programming
#8I was experimenting with how local, learnable routers can reduce token overhead, and lower costs, and decided to publish a post about it. The main goal is to delegate tool calls via a PyTorch based learner and examples of how to integrate this into a DSPy pipeline. Feedback welcome!
Can you put all of the code into a gist or something?
Re: Optimizing Tool Selection for LLM Workflows with Differentiable Programming
#9Yes I think once you’ve got an LLM in the loop it’s easy to be lazy and just use it to make all decisions. But it’s good to step back and think if there is a cheaper way, I mean even some hardcoded logic can do the job.
Re: Optimizing Tool Selection for LLM Workflows with Differentiable Programming
#10Yes I think once you’ve got an LLM in the loop it’s easy to be lazy and just use it to make all decisions. But it’s good to step back and think if there is a cheaper way, I mean even some hardcoded logic can do the job.
Very true. Making a non-deterministic system make determinations is also harder for it to do.
Right tool for the step to the right extent.
Feels like soft skills for software development.