Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Price per 1M tokens is meaningless

janilowski.pl

Post reply on HN
Print view View original

21–30 of 113 posts

Re: Price per 1M tokens is meaningless

#21

Post by greenavocado »

Jcampuzano2 wrote: post #19

I keep trying to convince directors and executives at my company to look past the cost per token amount but they refuse to do so. Those are the only things that actually give any sort of measurement of the monetary value of a token by these labs, and so its what many go by. For example there's some benchmarks that show that Opus for any task that requires a higher than `high` level of effort, may have actually been c…

That's why you have to reframe in terms of total cost per task and factor in model token generation quantity and multiply that by the base cost of the model. Then factor in your time value if you dare. Then you should get a more meaningful business metric.

Re: Price per 1M tokens is meaningless

#22

Post by sillysaurusx »

dbuxton wrote: post #12

As well as cost-per-task I think it's worth thinking about speed, especially in non-coding contexts that benchmark less cleanly We've started trying to do some comparison videos to capture more of the UX vs speed vs cost stuff e.g. https://www.linkedin.com/feed/update/urn:li:activity:7479891... which one of my team did for my LinkedIn account (disclaimer: marketing) (In this particular case Deepseek was way slower th…

Concrete example: I’ve been trying to use Claude to generate all my commit messages, but it takes 5-10x longer than if I just write them myself. Mine are less detailed, but one line changes are sometimes inconsequential (especially white space reformatting). I wish there was a model that understood the codebase well enough to generate commit messages in half the time.

Re: Price per 1M tokens is meaningless

#23

Post by teraflop »

yreg wrote: post #8

Cost per token doesn't say a lot, but "Cost per benchmark task" is also meaningless if your task is difficult enough that the cheaper model has no chance of cracking it.

That's not meaningless at all, it's a great metric! If the "cost per correctly-solved benchmark task" is infinity, you know not to use the model.

Re: Price per 1M tokens is meaningless

#24

Post by xienze »

Jcampuzano2 wrote: post #19

I keep trying to convince directors and executives at my company to look past the cost per token amount but they refuse to do so. Those are the only things that actually give any sort of measurement of the monetary value of a token by these labs, and so its what many go by. For example there's some benchmarks that show that Opus for any task that requires a higher than `high` level of effort, may have actually been c…

> may have actually been cheaper to use Fable on low even though the cost per token is drastically higher

Well that's the problem with these black boxes. You really have no idea beforehand how many tokens a given task is going to take. There's simply too many variables involved. It's therefore only natural for people to assume "the cheaper and older model is probably going to cost less overall to use than the newer, more expensive one."

Re: Price per 1M tokens is meaningless

#25

Post by nojito »

TacticalCoder wrote: post #15
nojito wrote: post #11

Earlier quoted context omitted.

>bring it in-house People don't like to hear this but the open models just aren't good for end to end agentic workflows. There are some very very good small open models that can excel in certain finite bounded tasks, but the foundational models are essential to building out agentic pipelines that actually work.

> People don't like to hear this but the open models just aren't good. Stuff like the latest DeepSeek, Kimchi and GLM are used and loved by many people. It's not using an open model that is difficult: it's having the hardware allowing to do so. It's pricey and require technical skills. That's why most people who are using (excellent btw) open-weight models are just renting compute online.

They just aren't good at agentic work.

Also risking it all for some distilled models is a recipe for disaster.

Re: Price per 1M tokens is meaningless

#26

Post by sillysaurusx »

efromvt wrote: post #20
yreg wrote: post #8

Cost per token doesn't say a lot, but "Cost per benchmark task" is also meaningless if your task is difficult enough that the cheaper model has no chance of cracking it.

Isn't the benchmark working exactly how it should in that case?

I was confused too. What they’re saying is, the average task you’re likely to do if you buy the model is the main predictor of costs. So if the average task benchmark is far higher than what you’re normally doing with it, you get a skewed perspective.

I’ve wanted a fast model to generate commit messages. No idea what that would be, but it doesn’t have to pass the SWE benchmarks very well. Just well enough that it understands the codebase.

Re: Price per 1M tokens is meaningless

#27

Post by sillysaurusx »

teraflop wrote: post #23
yreg wrote: post #8

Cost per token doesn't say a lot, but "Cost per benchmark task" is also meaningless if your task is difficult enough that the cheaper model has no chance of cracking it.

That's not meaningless at all, it's a great metric! If the "cost per correctly-solved benchmark task" is infinity, you know not to use the model.

They’re saying if the average task you actually use the model for is far less difficult than the benchmarks, you might incorrectly conclude that the model is costly when in fact it’s the best performing model for your actual use case.

I want a model that generates commit messages fast. Currently I have to wait up to a minute or two. That model doesn’t need to score very highly on SWE benchmarks, just highly enough that it can write out a good enough message in a few seconds. If you tested it on ${current top tier benchmark} you’d think it’s way too costly when in fact it’s the best tradeoff.

Re: Price per 1M tokens is meaningless

#28

Post by kpw94 »

In the context of local LLMs on limited hardware I've ran to the exact same conclusion: "tok/s" isn't the most useful metric when my personal North star metric, given my fixed hardware is: Model smart enough to execute my goals _in the minimum amount of time_.

Some models I tried (Mistral I think) had better tok/s, and roughly same billion parameters / scores on various benchmark... But they were _so_ verbose, that they generated many more tokens compared to a Qwen model of same caliber to answer the same thing.

So even though it had better generated tok/s, because so many more were generated, the clock time was longer.

And this compounds over mutli-turns: more generated token means more context used in the next turn (until some compaction or something runs)

Re: Price per 1M tokens is meaningless

#29

Post by pooploop64 »

janalsncm wrote: post #13

Pricing based on tokens always seemed a little weird to me.“Tokens” was and still is an engineering concept. The fundamental unit of transformer encoding and decoding. But I have a sinking feeling that many AI developers think “tokens” got their name from the same idea as “virtual tokens in a casino” which is more related to product pricing and business.

Tokens in a casino is pretty accurate if you think about it. You never really know what you'll get so it's tempting to "roll" over and over, thinking every roll puts you closer to a bellringer. It can even get addicting for some people.

Re: Price per 1M tokens is meaningless

#30

Post by [deleted] »

TacticalCoder wrote: post #15
nojito wrote: post #11

Earlier quoted context omitted.

>bring it in-house People don't like to hear this but the open models just aren't good for end to end agentic workflows. There are some very very good small open models that can excel in certain finite bounded tasks, but the foundational models are essential to building out agentic pipelines that actually work.

> People don't like to hear this but the open models just aren't good. Stuff like the latest DeepSeek, Kimchi and GLM are used and loved by many people. It's not using an open model that is difficult: it's having the hardware allowing to do so. It's pricey and require technical skills. That's why most people who are using (excellent btw) open-weight models are just renting compute online.

[deleted]
Post reply on HN