Earlier quoted context omitted.
LLMs don't have notions ... they are pattern matchers against a vast database of human text.
Please do a SELECT * from this database
What was the name of the rocket that brought the first humans into space?
Adaptive LLM routing under budget constraints
41–50 of 83 posts
Re: Adaptive LLM routing under budget constraints
#42Earlier quoted context omitted.
That and LLMs are seemingly plateauing. Earlier this year, it seemed like the big companies were releasing noticeable improvements every other week. People would joke a few weeks is “an eternity” in AI…so what time span are we looking at now?
There have been very large improvements in code generation in the last 6 months. A few weeks without improvement are not necessarily a plateau.
Wait until it ramps up so much that people will say "it's a plateau, for real this time" when they go 3 days without a +10% capability jump.
Re: Adaptive LLM routing under budget constraints
#43Earlier quoted context omitted.
So you don't expect AGI to be possible ever? Or is your concern mainly with the wildly different definitions people use for it and that we'll continue moving goal posts rather than agree we got there?
There's no concrete evidence AGI is possible mostly because it has no concrete definition. It's mostly hand waving, hype and credulity, and unproven claims of scalability right now. You can't move the goal posts because they don't exist.
Well, if a human is GI, we just need to make it Artificial. Easy.
Re: Adaptive LLM routing under budget constraints
#44> We instantiate this idea through Preference-prior Informed Linucb fOr adaptive rouTing (PILOT), a novel extension of LinUCB
Academics are pretty creative at naming their creations
Re: Adaptive LLM routing under budget constraints
#45Is this really the frontier of LLM research? I guess we really aren't getting AGI any time soon, then. It makes me a little less worried about the future, honestly. Edit: I never actually expected AGI from LLMs. That was snark. I just think it's notable that the fundamental gains in LLM performance seem to have dried up.
First, I don't think we will ever get to AGI. Not because we won't see huge advances still, but AGI is a moving ambiguous target that we won't get consensus on. But why does this paper impact your thinking on it? It is about budget and recognizing that different LLMs have different cost structures. It's not really an attempt to improve LLM performance measured absolutely.
I can totally see "it's not really AGI because it doesn't consistently outperform those three top 0.000001% outlier human experts yet if they work together".
It'll be a while until the ability to move the goalposts of "actual intelligence" is exhausted entirely.
Re: Adaptive LLM routing under budget constraints
#46Earlier quoted context omitted.
There have been very large improvements in code generation in the last 6 months. A few weeks without improvement are not necessarily a plateau.
Wait until it ramps up so much that people will say "it's a plateau, for real this time" when they go 3 days without a +10% capability jump.
I mean I wish there were a plateau, without one we're well onto our way into techno-feudalism. I just don't see it.
Re: Adaptive LLM routing under budget constraints
#47Earlier quoted context omitted.
"Do you think you need to do high/medium/low amount of thinking to answer X?" seems well within an LLMs wheelhouse if the goal is to build an optimized routing engine.
How do you think that an LLM could come by that information? Do you think that LLM vendors are logging performance and feeding that back into the model or some other mechanism?
Yes, that's why they keep getting better and why Anthropic is switching privacy policy defaults to eat my data please.
Re: Adaptive LLM routing under budget constraints
#48Earlier quoted context omitted.
"Do you think you need to do high/medium/low amount of thinking to answer X?" seems well within an LLMs wheelhouse if the goal is to build an optimized routing engine.
How do you think that an LLM could come by that information? Do you think that LLM vendors are logging performance and feeding that back into the model or some other mechanism?
Why not something dumb like this: https://chatgpt.com/share/68b60199-b6ac-8009-b50d-3e7cfff1d7... (gpt-4o)
Re: Adaptive LLM routing under budget constraints
#49Re: Adaptive LLM routing under budget constraints
#50GPT-4 at $24.7 per million tokens vs Mixtral at $0.24 - that's a 100x cost difference! Even if routing gets it wrong 20% of the time, the economics still work. But the real question is how you measure 'performance' - user satisfaction doesn't always correlate with technical metrics.
PPT (price-per-token) is insufficient to compute cost. You will also need to know an average tokens-per-interaction (TPI). They multiply to give you a cost estimate. A .01x PPT is wiped out by 100x TPI.
Are you saying that some models will take 100x more tokens than other (models in the same ballpark) for the same task? Is the 100 a real measured metric or just random numbers to illustrate a point?