Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Price per 1M tokens is meaningless

janilowski.pl

Post reply on HN
Print view View original

51–60 of 113 posts

Re: Price per 1M tokens is meaningless

#51

Post by cute_boi »

nathanyz wrote: post #16

The Sonnet 5 comment is spot on. Even Anthropic's own graph initially showed lower performance at higher costs. Only thing I notice about Sonnet 5 is that it does appear to hand off tasks to agents more frequently similar to Fable, but of course nowhere near the quality of Fable. My guess is that Opus 5 will do similar but just isn't ready yet.

Sonnet 5 is a huge regression and many times it performs worst than deepseek. I believe Antrophic staff themself don't use Sonnet and use Fable for everything.

Re: Price per 1M tokens is meaningless

#52

Post by lifeisstillgood »

lelandbatey wrote: post #43
nojito wrote: post #11

Earlier quoted context omitted.

>bring it in-house People don't like to hear this but the open models just aren't good for end to end agentic workflows. There are some very very good small open models that can excel in certain finite bounded tasks, but the foundational models are essential to building out agentic pipelines that actually work.

Have you been using those models? I've been using a hand-rolled orchestrator with Mimo v2.5 (I seem to be paying $0.017 per million/tokens after their heavy caching) and it's been very impressive. I started with it in Opencode as a harness, then had it build its own micro-harness with stdlib-only Python, then used that to build a local stdlib-only Orchestrator with CLI and web harness, and now I'm using that for impr…

So you are using a “cloud” provider and at 1c per million tokens …

Love to hear more about how you structure the orchestrator etc

Re: Price per 1M tokens is meaningless

#54

Post by bjt »

Lerc wrote: post #49

Cost per tokens is as valid as price per unit volume of fuel. Changing the fuel type, efficiency of your vehicle, driving distance, or driving conditions will all change how much it will cost you. Fuel cost per unit volume does not become meaningless just because you are neglecting all of the other factors involved. That would be throwing away the only data point you have been using. This is just asking for someone t…

Except, a gallon is a gallon no matter which gas station I'm at. Also I know my car's gas mileage, and it doesn't change when I visit a Shell station instead of a Chevron. The composition of the gas is regulated, as are the pumps that dispense it. There are inspectors from the state whose job it is to ensure that when I buy a gallon, I really get a gallon.

Tokenizers aren't standardized to anywhere near that level. A "token" from one isn't the same as a token from another.

Re: Price per 1M tokens is meaningless

#55

Post by sieabahlpark »

bjt wrote: post #54
Lerc wrote: post #49

Cost per tokens is as valid as price per unit volume of fuel. Changing the fuel type, efficiency of your vehicle, driving distance, or driving conditions will all change how much it will cost you. Fuel cost per unit volume does not become meaningless just because you are neglecting all of the other factors involved. That would be throwing away the only data point you have been using. This is just asking for someone t…

Except, a gallon is a gallon no matter which gas station I'm at. Also I know my car's gas mileage, and it doesn't change when I visit a Shell station instead of a Chevron. The composition of the gas is regulated, as are the pumps that dispense it. There are inspectors from the state whose job it is to ensure that when I buy a gallon, I really get a gallon. Tokenizers aren't standardized to anywhere near that level. A…

[dead]

Re: Price per 1M tokens is meaningless

#56

Post by kfsone »

I feel we are caught in a "this is fine, pay more and we may turn down the fire" situation.

The LLM itself produces one token. Some tool adds that token to the input and runs it again, flogging the horse. Downstream another tool, some kind of harness, tries to control this stream by injecting tokens into the context and then sending it to the inference tool, and then trying to pattern-match the output.

Finally, there you are on CodePorn.yata paying for an agent to generate code, paying for an agent to tell you what's wrong with it, and paying for an agent to make it differently bad, and hopefully move on to the next task.

If it still hasn't dawned on you that this isn't just a bubble, but a snake-oil-bubble-bath, just try to imagine the paradigm shift whereby you go on github.com, assign an issue to an agent, the agent fixes it by rewriting the application in Pascal but a reviewing agent catches that you wanted it to print a measurement in Pascals (pa), and you don't pay for the work or the review, you only pay for work that one or two reviewing agents determine is up to par.

Nobody is going to do that because as soon as they test it they're going to have to do some math that won't make sense without admitting/realizing it's not some near-sentient, AGI rating 0.9 intelligence, it's just a text prediction algorithm that can pull out entire sentences when you use it to infer output on topics it trained on.

Re: Price per 1M tokens is meaningless

#57

Post by Aurornis »

Lerc wrote: post #49

Cost per tokens is as valid as price per unit volume of fuel. Changing the fuel type, efficiency of your vehicle, driving distance, or driving conditions will all change how much it will cost you. Fuel cost per unit volume does not become meaningless just because you are neglecting all of the other factors involved. That would be throwing away the only data point you have been using. This is just asking for someone t…

That’s not a good analogy because a gallon of gasoline has a known amount of energy in it. The efficiency of each vehicle is also known, at least in a way that is easy to compare on a relative basis.

I can go to 10 different gas stations and buy the same amount of energy from them. When I put it in my car I’m going to get the same result out. The differences are very small.

Re: Price per 1M tokens is meaningless

#58

Post by dools »

It’s not meaningless at all: every query returns usage and I can calculate the cost.

EDIT: this is like saying hourly rate or salary is meaningless. Different people have different output. You have to evaluate performance.

EDIT2: just pray the LLM providers don’t start taking Patrick McKenzie’s advice and start charging based on “value delivered”

Re: Price per 1M tokens is meaningless

#59

Post by jst1fthsdys »

nojito wrote: post #25
TacticalCoder wrote: post #15

Earlier quoted context omitted.

> People don't like to hear this but the open models just aren't good. Stuff like the latest DeepSeek, Kimchi and GLM are used and loved by many people. It's not using an open model that is difficult: it's having the hardware allowing to do so. It's pricey and require technical skills. That's why most people who are using (excellent btw) open-weight models are just renting compute online.

They just aren't good at agentic work. Also risking it all for some distilled models is a recipe for disaster.

I disagree, they are that good at agentic work.

Re: Price per 1M tokens is meaningless

#60

Post by bArray »

The only metric that really matters is 'profit per amount invested'. This is very difficult to quickly evaluate, and therefore we resolve to use simplified metrics such as cost per unit tokens.

The point at which the metrics become meaningless is when others become aware of them, and begin to optimise for them. Lines per code is is not a bad insight for development activity, only when the developers are not aware of the metric. Price per 1M tokens became meaningless when LLM providers started to optimise for it. It seems to be that Sonnet 5 is optimised to score well on AA intelligence whilst seemingly having a lower price per 1M tokens.

I think generally we are in an AI bubble, and it will at some point pop. The numbers simply don't make sense. I would gamble heavily on local cost per task to survive the LLM winter. Given that hardware is pretty much a fixed overhead, you probably want to optimise for task per kW - that's where I'm betting.

Post reply on HN