Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Price per 1M tokens is meaningless

janilowski.pl

Post reply on HN
Print view View original

61–70 of 113 posts

Re: Price per 1M tokens is meaningless

#61

Post by efromvt »

yreg wrote: post #32
sillysaurusx wrote: post #26

Earlier quoted context omitted.

I was confused too. What they’re saying is, the average task you’re likely to do if you buy the model is the main predictor of costs. So if the average task benchmark is far higher than what you’re normally doing with it, you get a skewed perspective. I’ve wanted a fast model to generate commit messages. No idea what that would be, but it doesn’t have to pass the SWE benchmarks very well. Just well enough that it und…

Not even an average task. I can have a single task that I need to do and I could be choosing which model to use. The cheapest-per-benchmark-task model would be useless to me if it cannot do the task I need.

Exactly, so it has a success rate of 0 and infinite cost/completion on your relevant benchmark. If the benchmark doesn't map to what you need it to, then yeah, it's not a useful input.

Re: Price per 1M tokens is meaningless

#62

Post by [deleted] »

bArray wrote: post #60

The only metric that really matters is 'profit per amount invested'. This is very difficult to quickly evaluate, and therefore we resolve to use simplified metrics such as cost per unit tokens. The point at which the metrics become meaningless is when others become aware of them, and begin to optimise for them. Lines per code is is not a bad insight for development activity, only when the developers are not aware of…

[deleted]

Re: Price per 1M tokens is meaningless

#63

Post by c-hendricks »

kfsone wrote: post #56

I feel we are caught in a "this is fine, pay more and we may turn down the fire" situation. The LLM itself produces one token. Some tool adds that token to the input and runs it again, flogging the horse. Downstream another tool, some kind of harness, tries to control this stream by injecting tokens into the context and then sending it to the inference tool, and then trying to pattern-match the output. Finally, there…

One more ~~lane~~ layer of LLMs is sure to solve all our problems

Re: Price per 1M tokens is meaningless

#64

Post by nightpool »

sillysaurusx wrote: post #27
teraflop wrote: post #23

Earlier quoted context omitted.

That's not meaningless at all, it's a great metric! If the "cost per correctly-solved benchmark task" is infinity, you know not to use the model.

They’re saying if the average task you actually use the model for is far less difficult than the benchmarks, you might incorrectly conclude that the model is costly when in fact it’s the best performing model for your actual use case. I want a model that generates commit messages fast. Currently I have to wait up to a minute or two. That model doesn’t need to score very highly on SWE benchmarks, just highly enough th…

Your comment makes sense but I'm pretty sure yreg is saying the opposite of that—that their task is harder than the benchmark currently is.

(see their follow-up reply: "The cheapest-per-benchmark-task model would be useless to me if it cannot do the task I need.")

In either case, you need the right benchmark for the right task

Re: Price per 1M tokens is meaningless

#65

Post by paulddraper »

kfsone wrote: post #56

I feel we are caught in a "this is fine, pay more and we may turn down the fire" situation. The LLM itself produces one token. Some tool adds that token to the input and runs it again, flogging the horse. Downstream another tool, some kind of harness, tries to control this stream by injecting tokens into the context and then sending it to the inference tool, and then trying to pattern-match the output. Finally, there…

> it's just a text prediction algorithm that can pull out entire sentences when you use it to infer output on topics it trained on

This downplays the incredible things that can be done with it.

There's a lot of noise, yes. How long has the web existed? And yet we're still figuring out how to optimize (HTTP/3).

Disregard the signal at your own expense.

Re: Price per 1M tokens is meaningless

#66

Post by usef- »

Sonnet 5 makes more sense when you pretend the higher thinking efforts don't exist. (His test was on xhigh)

Anthropic's own release announcement mentioned that it's less cost competitive per task than Opus at higher thinking levels. It's significantly cheaper at lower levels though.

I'm wondering if this is going to be a universal pattern of smaller models: they're less smart, so to achieve the same benchmark results they have to think a lot more and hence become expensive.

Benchmarks force models to solve the problem entirely by themselves, requiring thinking. But if you pair them with a smart model (who thinks and solves beforehand) they won't need to solve the hard parts and can run on low/med. I suspect that was Anthropic's intention.

Re: Price per 1M tokens is meaningless

#67

Post by lelandbatey »

lifeisstillgood wrote: post #52
lelandbatey wrote: post #43

Earlier quoted context omitted.

Have you been using those models? I've been using a hand-rolled orchestrator with Mimo v2.5 (I seem to be paying $0.017 per million/tokens after their heavy caching) and it's been very impressive. I started with it in Opencode as a harness, then had it build its own micro-harness with stdlib-only Python, then used that to build a local stdlib-only Orchestrator with CLI and web harness, and now I'm using that for impr…

So you are using a “cloud” provider and at 1c per million tokens … Love to hear more about how you structure the orchestrator etc

Yes, it's a "cloud provider" but it's a cloud provider running an open model you can download (and that other cloud providers do host). I just happen to not have a computer big enough to host it.

As for the Orchestrator, it's pretty simple. In essence, it's like "Jira/Trello/Kanban on autopilot". Work items have states, a state machine defines how those work items transition between states, states are todo, in progress, retrying, reviewing, code reviewing, done. work items also have connections, allowing the LLMs to specify a dependency graph, and the dependency graph informs the dispatch order/parallelism, as well as when branches have to be merged. I talk to the steward, the steward has tool calls for interacting with all the data, and the orchestrator auto-dispatches all the work that comes in. I can generate work as fast as I can describe it to the steward, and that's usually the bottleneck.

So far I haven't had to deal with "how do you get the LLM to re-organize the work mid flight due to a worker finding something not accounted for by the planning", but I assume it'll come soon. The most complicated digraph I've tossed at it was 9 items and 4 layers deep. The kind of work I've given it hasn't been scoped large enough yet, so we'll see how it tackles that.

Re: Price per 1M tokens is meaningless

#68

Post by ttoinou »

dools wrote: post #58

It’s not meaningless at all: every query returns usage and I can calculate the cost. EDIT: this is like saying hourly rate or salary is meaningless. Different people have different output. You have to evaluate performance. EDIT2: just pray the LLM providers don’t start taking Patrick McKenzie’s advice and start charging based on “value delivered”

Can you really calculate the cost easily ? Given most of it should be reduced by input caching read (if you don’t want to have a crazy bill)

Re: Price per 1M tokens is meaningless

#69

Post by pier25 »

On top of that isn't it strange that if the LLM makes a mistake you're still charged for those tokens?

They're selling "intelligence", automation, etc but if the service doesn't work as expected the user has to pay for that.

Re: Price per 1M tokens is meaningless

#70

Post by EA-3167 »

paulddraper wrote: post #65
kfsone wrote: post #56

I feel we are caught in a "this is fine, pay more and we may turn down the fire" situation. The LLM itself produces one token. Some tool adds that token to the input and runs it again, flogging the horse. Downstream another tool, some kind of harness, tries to control this stream by injecting tokens into the context and then sending it to the inference tool, and then trying to pattern-match the output. Finally, there…

> it's just a text prediction algorithm that can pull out entire sentences when you use it to infer output on topics it trained on This downplays the incredible things that can be done with it. There's a lot of noise, yes. How long has the web existed? And yet we're still figuring out how to optimize (HTTP/3). Disregard the signal at your own expense.

What incredible things can be done with it?
Post reply on HN