Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Price per 1M tokens is meaningless

janilowski.pl

Post reply on HN
Print view View original

11–20 of 113 posts

Re: Price per 1M tokens is meaningless

#11

Post by nojito »

lifeisstillgood wrote: post #9

My advice to any CEO / individual - throw your hands in the air and bring it in-house. Yeah the performance can dip depending on what GPUs you can salvage these days but the uncertainty over price is almost nothing compared to the uncertainty over the effective use of AI. It’s not just coding (do I go partly agentic or all out Steve Yegge). This is all over the enterprise - do we parse every email, rewrite PowerPoint…

>bring it in-house

People don't like to hear this but the open models just aren't good for end to end agentic workflows.

There are some very very good small open models that can excel in certain finite bounded tasks, but the foundational models are essential to building out agentic pipelines that actually work.

Re: Price per 1M tokens is meaningless

#12

Post by dbuxton »

As well as cost-per-task I think it's worth thinking about speed, especially in non-coding contexts that benchmark less cleanly

We've started trying to do some comparison videos to capture more of the UX vs speed vs cost stuff e.g. https://www.linkedin.com/feed/update/urn:li:activity:7479891... which one of my team did for my LinkedIn account (disclaimer: marketing)

(In this particular case Deepseek was way slower than GPT 5.5 but I think that's because it installed Libreoffice half-way through the task!)

Re: Price per 1M tokens is meaningless

#13

Post by janalsncm »

Pricing based on tokens always seemed a little weird to me.“Tokens” was and still is an engineering concept. The fundamental unit of transformer encoding and decoding.

But I have a sinking feeling that many AI developers think “tokens” got their name from the same idea as “virtual tokens in a casino” which is more related to product pricing and business.

Re: Price per 1M tokens is meaningless

#14

Post by janalsncm »

yreg wrote: post #8

Cost per token doesn't say a lot, but "Cost per benchmark task" is also meaningless if your task is difficult enough that the cheaper model has no chance of cracking it.

Similarly, tasks that are too easy also aren’t ideal either. If a small model makes mistakes and backtracks but eventually cracks it, it will be using a lot more tokens than a bigger model that does it all with minimal mistakes.

Re: Price per 1M tokens is meaningless

#15

Post by TacticalCoder »

nojito wrote: post #11
lifeisstillgood wrote: post #9

My advice to any CEO / individual - throw your hands in the air and bring it in-house. Yeah the performance can dip depending on what GPUs you can salvage these days but the uncertainty over price is almost nothing compared to the uncertainty over the effective use of AI. It’s not just coding (do I go partly agentic or all out Steve Yegge). This is all over the enterprise - do we parse every email, rewrite PowerPoint…

>bring it in-house People don't like to hear this but the open models just aren't good for end to end agentic workflows. There are some very very good small open models that can excel in certain finite bounded tasks, but the foundational models are essential to building out agentic pipelines that actually work.

> People don't like to hear this but the open models just aren't good.

Stuff like the latest DeepSeek, Kimchi and GLM are used and loved by many people. It's not using an open model that is difficult: it's having the hardware allowing to do so. It's pricey and require technical skills.

That's why most people who are using (excellent btw) open-weight models are just renting compute online.

Re: Price per 1M tokens is meaningless

#16

Post by nathanyz »

The Sonnet 5 comment is spot on. Even Anthropic's own graph initially showed lower performance at higher costs. Only thing I notice about Sonnet 5 is that it does appear to hand off tasks to agents more frequently similar to Fable, but of course nowhere near the quality of Fable. My guess is that Opus 5 will do similar but just isn't ready yet.

Re: Price per 1M tokens is meaningless

#18

Post by sweetjuly »

janalsncm wrote: post #14
yreg wrote: post #8

Cost per token doesn't say a lot, but "Cost per benchmark task" is also meaningless if your task is difficult enough that the cheaper model has no chance of cracking it.

Similarly, tasks that are too easy also aren’t ideal either. If a small model makes mistakes and backtracks but eventually cracks it, it will be using a lot more tokens than a bigger model that does it all with minimal mistakes.

I think what you're really getting at is that it's only useful if the benchmarks are predictive of your workloads. If it predicts well (for example, your tasks are equally easy), then the fact that a larger model can complete it more quickly means that you may be able to complete the task more cheaply, depending on the token cost.

If the benchmarks are non-predictive, well, you can't use them for much of anything, which is of course a recurring problem with every benchmark ever.

Re: Price per 1M tokens is meaningless

#19

Post by Jcampuzano2 »

I keep trying to convince directors and executives at my company to look past the cost per token amount but they refuse to do so. Those are the only things that actually give any sort of measurement of the monetary value of a token by these labs, and so its what many go by.

For example there's some benchmarks that show that Opus for any task that requires a higher than `high` level of effort, may have actually been cheaper to use Fable on low even though the cost per token is drastically higher

Similarly with GPT 5.5 vs Opus. They simply look at the dollar amounts the labs assign to each model and run with it.

But part of the issue compounds on the fact that there are many people who simply default to the smartest model/effort and don't actually vary their model per task. So in some sense I don't actually blame them very much.

Post reply on HN