Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Adaptive LLM routing under budget constraints

arxiv.org

Post reply on HN
Print view View original

31–40 of 83 posts

Re: Adaptive LLM routing under budget constraints

#31

Post by nutjob2 »

nicce wrote: post #21
jibal wrote: post #13

Earlier quoted context omitted.

LLMs are not on the road to AGI, but there are plenty of dangers associated with them nonetheless.

Just 2 days ago Gemini 2.5 Pro tried to recommend me tax evasion based on non-existing laws and court decisions. The model was so charming and convincing, that even after I brought all the logic flaws and said that this is plain wrong, I started to doubt myself, because it is so good at pleasing, arguing and using words. And most would have accept the recommendation because the model sold it as less common tactic, wh…

Or you could understand the tool you are using and be skeptical of any of its output.

So many people just want to believe, instead of the reality of LLMs being quite unreliable.

Personally it's usually fairly obvious to me when LLMs are bullshitting probably because I have lots of experience detecting it in humans.

Re: Adaptive LLM routing under budget constraints

#32

Post by nutjob2 »

guluarte wrote: post #9
andrewflnr wrote: post #3

Is this really the frontier of LLM research? I guess we really aren't getting AGI any time soon, then. It makes me a little less worried about the future, honestly. Edit: I never actually expected AGI from LLMs. That was snark. I just think it's notable that the fundamental gains in LLM performance seem to have dried up.

I'm starting to think that there will not be an 'AGI' moment, we will simply slowly build smarter machines over time until we realize there is 'AGI'. It would be like video calls in the '90s everybody wanted them, now everybody hates them, lmao.

Or we'll realize that human intelligence and machine intelligence is apple and oranges.

Re: Adaptive LLM routing under budget constraints

#33

Post by simpaticoder »

pbd wrote: post #4

GPT-4 at $24.7 per million tokens vs Mixtral at $0.24 - that's a 100x cost difference! Even if routing gets it wrong 20% of the time, the economics still work. But the real question is how you measure 'performance' - user satisfaction doesn't always correlate with technical metrics.

PPT (price-per-token) is insufficient to compute cost. You will also need to know an average tokens-per-interaction (TPI). They multiply to give you a cost estimate. A .01x PPT is wiped out by 100x TPI.

Re: Adaptive LLM routing under budget constraints

#34

Post by mkoubaa »

pbd wrote: post #4

GPT-4 at $24.7 per million tokens vs Mixtral at $0.24 - that's a 100x cost difference! Even if routing gets it wrong 20% of the time, the economics still work. But the real question is how you measure 'performance' - user satisfaction doesn't always correlate with technical metrics.

> How you measure 'performance'

I heard the best way is through valuations

Re: Adaptive LLM routing under budget constraints

#35

Post by nutjob2 »

FINDarkside wrote: post #15
pbd wrote: post #4

GPT-4 at $24.7 per million tokens vs Mixtral at $0.24 - that's a 100x cost difference! Even if routing gets it wrong 20% of the time, the economics still work. But the real question is how you measure 'performance' - user satisfaction doesn't always correlate with technical metrics.

It's trivial to get better score than GPT-4 with 1% of the cost by using my propertiary routing algorithm that routes all requests to Gemini 2.5 Flash. It's called GASP (Gemini Always, Save Pennies)

Does anyone working in an individual capacity actually end up paying for Gemini (Flash or Pro)? Or does Google boil you like a frog and you end up subscribing?

Re: Adaptive LLM routing under budget constraints

#36

Post by muldvarp »

yahoozoo wrote: post #17
andrewflnr wrote: post #3

Is this really the frontier of LLM research? I guess we really aren't getting AGI any time soon, then. It makes me a little less worried about the future, honestly. Edit: I never actually expected AGI from LLMs. That was snark. I just think it's notable that the fundamental gains in LLM performance seem to have dried up.

That and LLMs are seemingly plateauing. Earlier this year, it seemed like the big companies were releasing noticeable improvements every other week. People would joke a few weeks is “an eternity” in AI…so what time span are we looking at now?

There have been very large improvements in code generation in the last 6 months. A few weeks without improvement are not necessarily a plateau.

Re: Adaptive LLM routing under budget constraints

#37

Post by nutjob2 »

fny wrote: post #20
delichon wrote: post #5

Earlier quoted context omitted.

> a strong enough notion of question complexity Aka Wisdom. No, LLMs don't have that. Me neither, I usually have to step in the rabbit holes in order to detect them.

"Do you think you need to do high/medium/low amount of thinking to answer X?" seems well within an LLMs wheelhouse if the goal is to build an optimized routing engine.

How do you think that an LLM could come by that information? Do you think that LLM vendors are logging performance and feeding that back into the model or some other mechanism?

Re: Adaptive LLM routing under budget constraints

#38

Post by aspect8445 »

nutjob2 wrote: post #35
FINDarkside wrote: post #15

Earlier quoted context omitted.

It's trivial to get better score than GPT-4 with 1% of the cost by using my propertiary routing algorithm that routes all requests to Gemini 2.5 Flash. It's called GASP (Gemini Always, Save Pennies)

Does anyone working in an individual capacity actually end up paying for Gemini (Flash or Pro)? Or does Google boil you like a frog and you end up subscribing?

I've used Gemini in a lot of personal projects. At this point I've probably made tens of thousands of requests, sometimes exceeding 1k per week. So far, I haven't had to pay a dime!

Re: Adaptive LLM routing under budget constraints

#39

Post by [deleted] »

nutjob2 wrote: post #37
fny wrote: post #20

Earlier quoted context omitted.

"Do you think you need to do high/medium/low amount of thinking to answer X?" seems well within an LLMs wheelhouse if the goal is to build an optimized routing engine.

How do you think that an LLM could come by that information? Do you think that LLM vendors are logging performance and feeding that back into the model or some other mechanism?

[deleted]

Re: Adaptive LLM routing under budget constraints

#40

Post by [deleted] »

yahoozoo wrote: post #17
andrewflnr wrote: post #3

Is this really the frontier of LLM research? I guess we really aren't getting AGI any time soon, then. It makes me a little less worried about the future, honestly. Edit: I never actually expected AGI from LLMs. That was snark. I just think it's notable that the fundamental gains in LLM performance seem to have dried up.

That and LLMs are seemingly plateauing. Earlier this year, it seemed like the big companies were releasing noticeable improvements every other week. People would joke a few weeks is “an eternity” in AI…so what time span are we looking at now?

[deleted]
Post reply on HN