Would there be advantages to routing to models according to cost in conjunction with prompt rewriting?
Adaptive LLM routing under budget constraints
51–60 of 83 posts
Re: Adaptive LLM routing under budget constraints
#52Earlier quoted context omitted.
It's trivial to get better score than GPT-4 with 1% of the cost by using my propertiary routing algorithm that routes all requests to Gemini 2.5 Flash. It's called GASP (Gemini Always, Save Pennies)
Does anyone working in an individual capacity actually end up paying for Gemini (Flash or Pro)? Or does Google boil you like a frog and you end up subscribing?
I've paid a few dollars a month for my API usage for about 6 months.
Re: Adaptive LLM routing under budget constraints
#53Earlier quoted context omitted.
Just 2 days ago Gemini 2.5 Pro tried to recommend me tax evasion based on non-existing laws and court decisions. The model was so charming and convincing, that even after I brought all the logic flaws and said that this is plain wrong, I started to doubt myself, because it is so good at pleasing, arguing and using words. And most would have accept the recommendation because the model sold it as less common tactic, wh…
Or you could understand the tool you are using and be skeptical of any of its output. So many people just want to believe, instead of the reality of LLMs being quite unreliable. Personally it's usually fairly obvious to me when LLMs are bullshitting probably because I have lots of experience detecting it in humans.
LLM is only useful if it gives shortcut to information with reasonable accuracy. If I need to double check everything, it is just extra step.
In this case I just happened to be domain expert and knew it was wrong. It would have required significant effort to verify everything with some less experienced person.
Re: Adaptive LLM routing under budget constraints
#54These router papers are popping up hard now. I have a gradient boosted router I've been playing with that ties into retrieval to provide adaptive routing. The truth about these routers is that you have to tune them on your workloads to get the full benefit, otherwise they test way better than they work in production. That was why I added the retrieval aspect to mine, otherwise your top line slice and reality are very different.
Re: Adaptive LLM routing under budget constraints
#55Earlier quoted context omitted.
Just 2 days ago Gemini 2.5 Pro tried to recommend me tax evasion based on non-existing laws and court decisions. The model was so charming and convincing, that even after I brought all the logic flaws and said that this is plain wrong, I started to doubt myself, because it is so good at pleasing, arguing and using words. And most would have accept the recommendation because the model sold it as less common tactic, wh…
> even after I brought all the logic flaws and said that this is plain wrong Once you've started to argue with an LLM you're already barking up the wrong tree. Maybe you're right, maybe not, but there's no point in arguing it out with an LLM.
There are cases when they are actually correct, instead of the human.
Re: Adaptive LLM routing under budget constraints
#56> We instantiate this idea through Preference-prior Informed Linucb fOr adaptive rouTing (PILOT), a novel extension of LinUCB Academics are pretty creative at naming their creations
I almost named my LoRA replacement BEMO, but that felt too cute, so it's just BEM (Bolt-on Expert Modules).
Re: Adaptive LLM routing under budget constraints
#57Earlier quoted context omitted.
> even after I brought all the logic flaws and said that this is plain wrong Once you've started to argue with an LLM you're already barking up the wrong tree. Maybe you're right, maybe not, but there's no point in arguing it out with an LLM.
There are cases when they are actually correct, instead of the human.
Yes, and there's a substantial chance they'll apologize to you anyway even when they were right. There's no reason to expect them to be more likely to apologize when they're actually right vs actually wrong- their agreeableness is really orthogonal to their correctness.
Re: Adaptive LLM routing under budget constraints
#58Incredible that they are using contextual bandits, and named it: Preference-prior Informed Linucb fOr adaptive rouTing (PILOT) Rather than the much more obvious: Preference-prior Informed Linucb For Adaptive Routing (PILFAR)
That's pretty funny. I might need to pilfer it.
Re: Adaptive LLM routing under budget constraints
#59Earlier quoted context omitted.
There are cases when they are actually correct, instead of the human.
Yes, and there's a substantial chance they'll apologize to you anyway even when they were right. There's no reason to expect them to be more likely to apologize when they're actually right vs actually wrong- their agreeableness is really orthogonal to their correctness.
Yes, they over-apologize. But my main reason for using LLMs is seeking out things that I missed myself or my own argumentation was not good. Sometimes they are really good at bringing new perspectives. Whether they are correct or incorrect is not the point - are they giving argument or perspective that is worth inspecting more with my own brains?
Re: Adaptive LLM routing under budget constraints
#60Earlier quoted context omitted.
PPT (price-per-token) is insufficient to compute cost. You will also need to know an average tokens-per-interaction (TPI). They multiply to give you a cost estimate. A .01x PPT is wiped out by 100x TPI.
Are you saying that some models will take 100x more tokens than other (models in the same ballpark) for the same task? Is the 100 a real measured metric or just random numbers to illustrate a point?
With thinking models, yes 100x is not just possible, but probable. You get charged for the intermediate thinking tokens, even if you don't see them (which is the case for Grok, for example). And even if you do see them, they won't necessarily add value.