Incredible that they are using contextual bandits, and named it:
Preference-prior Informed Linucb fOr adaptive rouTing (PILOT)
Rather than the much more obvious: Preference-prior Informed Linucb For Adaptive Routing (PILFAR)
Adaptive LLM routing under budget constraints
11–20 of 83 posts
Re: Adaptive LLM routing under budget constraints
#12Is there a reason human preference data is even needed? Don't LLMs already have a strong enough notion of question complexity to build a dataset for routing?
LLMs don't have notions ... they are pattern matchers against a vast database of human text.
Re: Adaptive LLM routing under budget constraints
#13Is this really the frontier of LLM research? I guess we really aren't getting AGI any time soon, then. It makes me a little less worried about the future, honestly. Edit: I never actually expected AGI from LLMs. That was snark. I just think it's notable that the fundamental gains in LLM performance seem to have dried up.
LLMs are not on the road to AGI, but there are plenty of dangers associated with them nonetheless.
Re: Adaptive LLM routing under budget constraints
#14GPT-4 at $24.7 per million tokens vs Mixtral at $0.24 - that's a 100x cost difference! Even if routing gets it wrong 20% of the time, the economics still work. But the real question is how you measure 'performance' - user satisfaction doesn't always correlate with technical metrics.
> GPT-4 at $24.7 per million tokens
While technically true why would you want to use it when OpenAI itself provides a bunch of many times cheaper and better models?
Re: Adaptive LLM routing under budget constraints
#15GPT-4 at $24.7 per million tokens vs Mixtral at $0.24 - that's a 100x cost difference! Even if routing gets it wrong 20% of the time, the economics still work. But the real question is how you measure 'performance' - user satisfaction doesn't always correlate with technical metrics.
It's trivial to get better score than GPT-4 with 1% of the cost by using my propertiary routing algorithm that routes all requests to Gemini 2.5 Flash. It's called GASP (Gemini Always, Save Pennies)
Re: Adaptive LLM routing under budget constraints
#16Is there a reason human preference data is even needed? Don't LLMs already have a strong enough notion of question complexity to build a dataset for routing?
LLMs don't have notions ... they are pattern matchers against a vast database of human text.
Please do a SELECT * from this database
Re: Adaptive LLM routing under budget constraints
#17Is this really the frontier of LLM research? I guess we really aren't getting AGI any time soon, then. It makes me a little less worried about the future, honestly. Edit: I never actually expected AGI from LLMs. That was snark. I just think it's notable that the fundamental gains in LLM performance seem to have dried up.
That and LLMs are seemingly plateauing. Earlier this year, it seemed like the big companies were releasing noticeable improvements every other week. People would joke a few weeks is “an eternity” in AI…so what time span are we looking at now?
Re: Adaptive LLM routing under budget constraints
#18The framing in the headline is interesting. As far as I recall, spending 4x more compute on a model to improve performance by 7% is the move that has worked over and over again up to this point. 101 % of GPT-4 performance (potentially at any cost) is what I would expect an improved routing algorithm to achieve.
(The submitted title was "93% of GPT-4 performance at 1/4 cost: LLM routing with weak bandit feedback")
Re: Adaptive LLM routing under budget constraints
#19Is this really the frontier of LLM research? I guess we really aren't getting AGI any time soon, then. It makes me a little less worried about the future, honestly. Edit: I never actually expected AGI from LLMs. That was snark. I just think it's notable that the fundamental gains in LLM performance seem to have dried up.
just because it’s on arxiv doesn’t mean anything
arxiv is essentially a blog under an academic format, popular amongst asian and south asian academic communities
currently you can launder reputation with it, just like “white papers” in the crypto world allowed for capital for some time
this ability will diminish as more people catch on
Re: Adaptive LLM routing under budget constraints
#20Is there a reason human preference data is even needed? Don't LLMs already have a strong enough notion of question complexity to build a dataset for routing?
> a strong enough notion of question complexity Aka Wisdom. No, LLMs don't have that. Me neither, I usually have to step in the rabbit holes in order to detect them.
"Do you think you need to do high/medium/low amount of thinking to answer X?" seems well within an LLMs wheelhouse if the goal is to build an optimized routing engine.