[flagged]
Price per 1M tokens is meaningless
91–100 of 113 posts
Re: Price per 1M tokens is meaningless
#92I feel we are caught in a "this is fine, pay more and we may turn down the fire" situation. The LLM itself produces one token. Some tool adds that token to the input and runs it again, flogging the horse. Downstream another tool, some kind of harness, tries to control this stream by injecting tokens into the context and then sending it to the inference tool, and then trying to pattern-match the output. Finally, there…
[flagged]
>Careful you don't "it's just a text predictor" yourself into unemployment .
Says the guy who is "I-can-now-just-push-buttons-and-get-paid-forever"-ing into not only into unemployment, but also being totally redundant.
Before you come back with, "Oh I review everything LLM does": Only in your dreams you ll gain or retain experience by just reviewing stuff. That is even if you actually review every line that the LLM writes...which have literally zero chance of happening. People have hard time keeping their eye open even when their life is on the line when using a pseudo self driving car...
Re: Price per 1M tokens is meaningless
#93Not every application uses LLMs the same way. For some use-cases, price per 1M tokens is absolutely meaningful. Eg, we do a lot of pretty basic classification/entity-extraction/summarization type work on large inputs (100k tokens per request being very common). It's pretty easy stuff; Gemini 2.0 Flash was perfectly adequate, quite fast, and cost $0.10 per 1M tokens (and even less when we could make use of the batch API). Every newer more powerful model obviously can handle the same work but costs significantly more. When we're deciding what model to use, price per 1M tokens is definitely a meaningful metric.
Re: Price per 1M tokens is meaningless
#94Related to this, for our use case, setting thinking to high instead of low made tasks complete faster and cheaper (Gemini 3.0 flash). Other aspects are caching, often at 0.1X cost, where providers really differ in how efficient they are (Anthropic really good, Google not so much) and how chatty a model is (costing output tokens).
I also don't see much of a quality difference switching between thinking levels while using those models as agents.
Re: Price per 1M tokens is meaningless
#95cost per benchmark task is definitely interesting! i've always wanted cost per prompt, but even that has too much variation.
Well yeah this benchmark I've quoted is quite large, I believe they use thousands of diverse tasks, so this average cost per task could be a more accurate representation of how expensive each model actually is to run.
Re: Price per 1M tokens is meaningless
#96Price per token is meaningless for more reasons than this, because all of the provider monthly subscriptions price tokens _extremely_ differently than their per-token billing rates. It's stupid to look only at what you get when paying more than you need to for a given service.
Yes that's true, although calculating the exact discount you get using a subscription could be difficult, especially since the labs often don't state their actual limits and just tell you you get "more usage" or "even more usage" or "5x the usage" (but how much exactly is x?).
It is clear we are getting a very meaningful discount by using a subscription though. I once checked how much would I pay for Codex after about 3 months of non-daily use if I only bought "credits". Couldn't believe it when it came out to be north of $1000 (I paid them $20/month in that period).
Re: Price per 1M tokens is meaningless
#97Pricing based on tokens always seemed a little weird to me.“Tokens” was and still is an engineering concept. The fundamental unit of transformer encoding and decoding. But I have a sinking feeling that many AI developers think “tokens” got their name from the same idea as “virtual tokens in a casino” which is more related to product pricing and business.
Tokens do reflect the provider's cost though - each token output required them to execute the model once, normally incurring a fixed amount of compute per token.
I am not sure if pricing actually reflects inference costs
Re: Price per 1M tokens is meaningless
#98An LLM is an extremely complex thing used for all manner of purposes. The hope that there would be some simple pricing construct that would map nicely to value provided is a pipe dream. Pricing per token is at least reasonably straight forward. If you aren't getting value, you don't use the service. One doesn't buy a Ferrari and then complain that in their town Ferrari doesn't help them pick up women and hence it sho…
I'd say it's more like going to a Ferrari dealership and they tell you they will build a car for you and bill you per gram of parts used. It might also not work. And they might also not build it ever, really — but they will bill you for any attempts to build it.
Re: Price per 1M tokens is meaningless
#99Another important benchmark would be — cost per benchmark task using subscription tokens. Since most of us are using subscriptions and cost per token there is quite different from API costs.
Re: Price per 1M tokens is meaningless
#100Price per token is meaningless for more reasons than this, because all of the provider monthly subscriptions price tokens _extremely_ differently than their per-token billing rates. It's stupid to look only at what you get when paying more than you need to for a given service.
Yes that's true, although calculating the exact discount you get using a subscription could be difficult, especially since the labs often don't state their actual limits and just tell you you get "more usage" or "even more usage" or "5x the usage" (but how much exactly is x?). It is clear we are getting a very meaningful discount by using a subscription though. I once checked how much would I pay for Codex after abou…
> it came out to be north of $1000 (I paid them $20/month in that period)
Yeah. My $200/month subscription would easily cost $5000/month or more if I were paying per token.