Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Price per 1M tokens is meaningless

janilowski.pl

Post reply on HN
Print view View original

71–80 of 113 posts

Re: Price per 1M tokens is meaningless

#71

Post by wccrawford »

pier25 wrote: post #69

On top of that isn't it strange that if the LLM makes a mistake you're still charged for those tokens? They're selling "intelligence", automation, etc but if the service doesn't work as expected the user has to pay for that.

You're paying for a service with known flaws. They do not guarantee correct answers.

Also, LLMs don't "make mistakes". They don't think or act. Every single thing they output is a hallucination. It just happens that the vast majority of things align with reality.

Re: Price per 1M tokens is meaningless

#72

Post by Aeolun »

pier25 wrote: post #69

On top of that isn't it strange that if the LLM makes a mistake you're still charged for those tokens? They're selling "intelligence", automation, etc but if the service doesn't work as expected the user has to pay for that.

If I use electricity to do something stupid, I still have to pay for the electricity. Intelligence is just another utility.

Re: Price per 1M tokens is meaningless

#73

Post by SwellJoe »

Efficiency is the next frontier in LLMs, and I'm not confident the American companies are taking it seriously enough. DeepSeek, even in a naive API-calling loop, serves something like 80-90% cached tokens at an absurdly low price per token. Using an agent harness tuned specifically for their caching (Reasonix) pushes the cached tokens to 97-99%. DeepSeek is consistently among the cheapest models per task in my benchmarks, while also performing quite well. I'm still almost always using Claude for work, but for side projects, small stuff, etc. and anything better served by an API rather than starting up Claude Code (or `claude -p`) I'm using DeepSeek pretty often.

Anthropic models also shut down on a lot of security-related work, which is what I've been spending a lot of time on lately. I expected Fable to refuse this kind of task, but even Opus 4.8 refuses to build a verification harness for security bugs, as that involves exercising a discovered bug to prove it's been fixed in an automated red/green way, which looks like exploit creation to Opus' guardrails. So, I have to use other models for that work, now, though most of the original benchmarks I built were built with Claude.

Re: Price per 1M tokens is meaningless

#74

Post by budsniffer952 »

kfsone wrote: post #56

I feel we are caught in a "this is fine, pay more and we may turn down the fire" situation. The LLM itself produces one token. Some tool adds that token to the input and runs it again, flogging the horse. Downstream another tool, some kind of harness, tries to control this stream by injecting tokens into the context and then sending it to the inference tool, and then trying to pattern-match the output. Finally, there…

[flagged]

Re: Price per 1M tokens is meaningless

#75

Post by kpw94 »

c7b wrote: post #39
kpw94 wrote: post #28

In the context of local LLMs on limited hardware I've ran to the exact same conclusion: "tok/s" isn't the most useful metric when my personal North star metric, given my fixed hardware is: Model smart enough to execute my goals _in the minimum amount of time_. Some models I tried (Mistral I think) had better tok/s, and roughly same billion parameters / scores on various benchmark... But they were _so_ verbose, that t…

Even more important in a local context is the difference between token generation and prompt processing speed. We tend to focus on the former, but for multi-turn/agentic workflows the latter can dominate.

Yeah definitely. I've recently commented on that: https://news.ycombinator.com/item?id=48557890

Re: Price per 1M tokens is meaningless

#76

Post by dools »

ttoinou wrote: post #68
dools wrote: post #58

It’s not meaningless at all: every query returns usage and I can calculate the cost. EDIT: this is like saying hourly rate or salary is meaningless. Different people have different output. You have to evaluate performance. EDIT2: just pray the LLM providers don’t start taking Patrick McKenzie’s advice and start charging based on “value delivered”

Can you really calculate the cost easily ? Given most of it should be reduced by input caching read (if you don’t want to have a crazy bill)

Yeah you get the usage back and each element has pricing published

Re: Price per 1M tokens is meaningless

#77

Post by budsniffer952 »

EA-3167 wrote: post #70
paulddraper wrote: post #65

Earlier quoted context omitted.

> it's just a text prediction algorithm that can pull out entire sentences when you use it to infer output on topics it trained on This downplays the incredible things that can be done with it. There's a lot of noise, yes. How long has the web existed? And yet we're still figuring out how to optimize (HTTP/3). Disregard the signal at your own expense.

What incredible things can be done with it?

[flagged]

Re: Price per 1M tokens is meaningless

#78

Post by asdewqqwer »

cute_boi wrote: post #51
nathanyz wrote: post #16

The Sonnet 5 comment is spot on. Even Anthropic's own graph initially showed lower performance at higher costs. Only thing I notice about Sonnet 5 is that it does appear to hand off tasks to agents more frequently similar to Fable, but of course nowhere near the quality of Fable. My guess is that Opus 5 will do similar but just isn't ready yet.

Sonnet 5 is a huge regression and many times it performs worst than deepseek. I believe Antrophic staff themself don't use Sonnet and use Fable for everything.

Given the capability of fable and the shockingly repetitive silly mistakes they made when publishing/updating something, I am starting to wondering whether Anthrophic can afford Fable for everything themselves.

Re: Price per 1M tokens is meaningless

#79

Post by thfuran »

sillysaurusx wrote: post #27
teraflop wrote: post #23

Earlier quoted context omitted.

That's not meaningless at all, it's a great metric! If the "cost per correctly-solved benchmark task" is infinity, you know not to use the model.

They’re saying if the average task you actually use the model for is far less difficult than the benchmarks, you might incorrectly conclude that the model is costly when in fact it’s the best performing model for your actual use case. I want a model that generates commit messages fast. Currently I have to wait up to a minute or two. That model doesn’t need to score very highly on SWE benchmarks, just highly enough th…

As always, the relevance of any given benchmark depends on how similar what it’s testing is to your workload.
Post reply on HN