would be nice to have these benchmarks so they can be run against models like the qwen family, gemini, etc.
Price per 1M tokens is meaningless
31–40 of 113 posts
Re: Price per 1M tokens is meaningless
#32Earlier quoted context omitted.
Isn't the benchmark working exactly how it should in that case?
I was confused too. What they’re saying is, the average task you’re likely to do if you buy the model is the main predictor of costs. So if the average task benchmark is far higher than what you’re normally doing with it, you get a skewed perspective. I’ve wanted a fast model to generate commit messages. No idea what that would be, but it doesn’t have to pass the SWE benchmarks very well. Just well enough that it und…
Not even an average task. I can have a single task that I need to do and I could be choosing which model to use. The cheapest-per-benchmark-task model would be useless to me if it cannot do the task I need.
Re: Price per 1M tokens is meaningless
#33My advice to any CEO / individual - throw your hands in the air and bring it in-house. Yeah the performance can dip depending on what GPUs you can salvage these days but the uncertainty over price is almost nothing compared to the uncertainty over the effective use of AI. It’s not just coding (do I go partly agentic or all out Steve Yegge). This is all over the enterprise - do we parse every email, rewrite PowerPoint…
>bring it in-house People don't like to hear this but the open models just aren't good for end to end agentic workflows. There are some very very good small open models that can excel in certain finite bounded tasks, but the foundational models are essential to building out agentic pipelines that actually work.
>> just aren't good for end to end agentic workflows.
This is (apparently) the conceit of SteveYegge / GasTown - no model can cope unassisted so chunk it up, run it and if it falls over remember the exact place and restart, merging it all in
But that’s not my point.
I believe that software is a new form of literacy and just as all Companies and societies are literate now, in the future (tm) companies will run exclusively on software - AI developed software and those who go all out will have the sort of advantages the Catholic Church had over .. guilds?
Anyhow, that’s me being AI optimist. But writing the code is going to be a small part of that transition - almost everything to do with LLMs that is claimed amazing (Computer vision is something else) - almost everything people say we need an LLM is stuff you could have done three years ago but your internal politics just would not let you. Oh look we can see if our policies are being met (you could have written the policies in code and solved the whole problem)
Im struggling to get it out but - almost everything AI is proposed for is stuff a well run engineering firm coukd have taken on. A software literate firm could have done without AI is where firms are hoping AI will Get them
Imagine how far ahead real software literate firms will be - as long as they don’t burn their runway in tokens
Which is why, the right play imo is still buy in-house as much as possible, engineer around the problems and explore the phase space at marginal Cost.
Then and only then think frontier models.
Re: Price per 1M tokens is meaningless
#34Earlier quoted context omitted.
Similarly, tasks that are too easy also aren’t ideal either. If a small model makes mistakes and backtracks but eventually cracks it, it will be using a lot more tokens than a bigger model that does it all with minimal mistakes.
I think what you're really getting at is that it's only useful if the benchmarks are predictive of your workloads. If it predicts well (for example, your tasks are equally easy), then the fact that a larger model can complete it more quickly means that you may be able to complete the task more cheaply, depending on the token cost. If the benchmarks are non-predictive, well, you can't use them for much of anything, wh…
Yeah, if the benchmark is actually predictive of the tasks you have then it is trivial to conclude that the cheapest-per-benchmark-task model will be the cheapest one for your tasks…
Re: Price per 1M tokens is meaningless
#35Earlier quoted context omitted.
> People don't like to hear this but the open models just aren't good. Stuff like the latest DeepSeek, Kimchi and GLM are used and loved by many people. It's not using an open model that is difficult: it's having the hardware allowing to do so. It's pricey and require technical skills. That's why most people who are using (excellent btw) open-weight models are just renting compute online.
They just aren't good at agentic work. Also risking it all for some distilled models is a recipe for disaster.
GLM 5.2 is the first model that is competing with the frontier, everything before it existed I would totally agree with you.
Re: Price per 1M tokens is meaningless
#36Pricing based on tokens always seemed a little weird to me.“Tokens” was and still is an engineering concept. The fundamental unit of transformer encoding and decoding. But I have a sinking feeling that many AI developers think “tokens” got their name from the same idea as “virtual tokens in a casino” which is more related to product pricing and business.
Tokens do reflect the provider's cost though - each token output required them to execute the model once, normally incurring a fixed amount of compute per token.
Re: Price per 1M tokens is meaningless
#37[dead]
Re: Price per 1M tokens is meaningless
#38Earlier quoted context omitted.
I was confused too. What they’re saying is, the average task you’re likely to do if you buy the model is the main predictor of costs. So if the average task benchmark is far higher than what you’re normally doing with it, you get a skewed perspective. I’ve wanted a fast model to generate commit messages. No idea what that would be, but it doesn’t have to pass the SWE benchmarks very well. Just well enough that it und…
Not even an average task. I can have a single task that I need to do and I could be choosing which model to use. The cheapest-per-benchmark-task model would be useless to me if it cannot do the task I need.
Routing tasks to models by complexity like a job for a LLM.
I'm sure there are degenerate cases, but I'd bet a relatively small model could do the job.
Re: Price per 1M tokens is meaningless
#39In the context of local LLMs on limited hardware I've ran to the exact same conclusion: "tok/s" isn't the most useful metric when my personal North star metric, given my fixed hardware is: Model smart enough to execute my goals _in the minimum amount of time_. Some models I tried (Mistral I think) had better tok/s, and roughly same billion parameters / scores on various benchmark... But they were _so_ verbose, that t…
Even more important in a local context is the difference between token generation and prompt processing speed. We tend to focus on the former, but for multi-turn/agentic workflows the latter can dominate.
Re: Price per 1M tokens is meaningless
#40My advice to any CEO / individual - throw your hands in the air and bring it in-house. Yeah the performance can dip depending on what GPUs you can salvage these days but the uncertainty over price is almost nothing compared to the uncertainty over the effective use of AI. It’s not just coding (do I go partly agentic or all out Steve Yegge). This is all over the enterprise - do we parse every email, rewrite PowerPoint…
>bring it in-house People don't like to hear this but the open models just aren't good for end to end agentic workflows. There are some very very good small open models that can excel in certain finite bounded tasks, but the foundational models are essential to building out agentic pipelines that actually work.
>end to end agentic workflows.
There are probably 100 competing versions what this phrase might encapsulate. Could you elaborate more on which version you are using exactly?
My experience is that frontier models are only marginally better and not close to the cost/value of the open models which are anywhere from 10-100x cheaper. Perhaps I'm not doing "end to end agentic workflows?"