Adaptive LLM routing under budget constraints • Hacker News Board

Re: Adaptive LLM routing under budget constraints

#81

Post by datadrivenangel » Tue, Sep 02, 2025, 1:10 PM UTC

monsieurbanana wrote: post #50

simpaticoder wrote: post #33

Earlier quoted context omitted.

PPT (price-per-token) is insufficient to compute cost. You will also need to know an average tokens-per-interaction (TPI). They multiply to give you a cost estimate. A .01x PPT is wiped out by 100x TPI.

Are you saying that some models will take 100x more tokens than other (models in the same ballpark) for the same task? Is the 100 a real measured metric or just random numbers to illustrate a point?

the GPT 5 models use ~10x more tokens depending on the reasoning settings.

Re: Adaptive LLM routing under budget constraints

#82

Post by neuronexmachina » Tue, Sep 02, 2025, 2:18 PM UTC

9dev wrote: post #61

ACCount37 wrote: post #45

Earlier quoted context omitted.

I can totally see "it's not really AGI because it doesn't consistently outperform those three top 0.000001% outlier human experts yet if they work together". It'll be a while until the ability to move the goalposts of "actual intelligence" is exhausted entirely.

Well right now, my niece of 7 years outperforms all LLM contenders in drawing a Pelican on a bicycle

I tried it in Gemini just now, it seems to have done a decent job: https://g.co/gemini/share/b6fef8398c01

Re: Adaptive LLM routing under budget constraints

#83

Post by monsieurbanana » Wed, Sep 03, 2025, 8:37 AM UTC

simpaticoder wrote: post #60

monsieurbanana wrote: post #50

Earlier quoted context omitted.

Are you saying that some models will take 100x more tokens than other (models in the same ballpark) for the same task? Is the 100 a real measured metric or just random numbers to illustrate a point?

With thinking models, yes 100x is not just possible, but probable. You get charged for the intermediate thinking tokens, even if you don't see them (which is the case for Grok, for example). And even if you do see them, they won't necessarily add value.

> With thinking models, yes 100x is not just possible, but probable

So the answer is no then, because I don't put reasoning and non-reasoning models in the same ballpark when it comes to token usage. You can just turn off reasoning.