Yes, and verbosity (thinking) is a huge factor.
The real prices of frontier models
21–30 of 91 posts
Re: The real prices of frontier models
#22Honestly, I find performant pricing where they test each model on the same task is much more useful than figuring out the tokens or using the input token tax.
The reason being is that the only tokens I feel I really control are the input tokens, but the whole program seems to just run itself and they just charge you what they want to charge you and it’s more of a black box.
Very interesting article though.
Re: The real prices of frontier models
#23Yeah, Anthropic's current tokenizer in Sonnet 5/Opus 4.8/Fable 5 is much worse than OpenAI's. Also, OpenAI has been using their current o200k_base from the day GPT-4o came out over two years ago. Just a few of my own tests: - A ~2000-2002 legacy C++ game codebase at about ~90kloc: GPT 1.12M, Claude 2.2M - A ~30kloc TypeScript codebase: GPT 260K, Claude 437K In the end, GPT's current tokenizer is ~1.6x-2x better than…
you use the wrong word
the Anthropic tokenizer is not worse, its more expensive/verbose
Re: The real prices of frontier models
#24This piece focuses on the cost differences from the tokenizer, which do matter, but I wish they emphasized more that even adding the tokenizer to your calculation doesn't provide you with a good way to calculate cost for agentic coding tasks. Other traits where models differ that have an even greater impact on your total spend: * How much context do they load in to solve a given task? * How long do they spend thinkin…
[deleted]
Re: The real prices of frontier models
#25We've switched the default model in playcode.io among Opus 4.8, Opus 4.6, Sonnet 4.6, and Sonnet 5. I must admit, Opus 4.8 is quite expensive, and the costs accumulate quickly. Opus 4.6 is about 50% cheaper, while Sonnet 5 is significantly more affordable. According to the data, Sonnet 5 is about 2-3 times cheaper. Fable 5 is unaffordable at all...
Today, I tested Sol 5.6 on various tasks. It performs similarly to Opus 4.8 but is still noticeably more expensive than Sonnet 5. Although Sonnet 5 isn't the top model, it's quite effective for creating typical websites for small and medium businesses. However, they will increase the price starting September 1, as their free offer is ending.
I'm also actively testing Grok 4.5. There's something promising about it. The design is mediocre, in my opinion, but it operates quickly and reliably without any deadloops. Usually, Grok models would fail or loop, but this one is stable.
Overall, I really want a benchmark based on real tasks.
Re: The real prices of frontier models
#26Yeah, Anthropic's current tokenizer in Sonnet 5/Opus 4.8/Fable 5 is much worse than OpenAI's. Also, OpenAI has been using their current o200k_base from the day GPT-4o came out over two years ago. Just a few of my own tests: - A ~2000-2002 legacy C++ game codebase at about ~90kloc: GPT 1.12M, Claude 2.2M - A ~30kloc TypeScript codebase: GPT 260K, Claude 437K In the end, GPT's current tokenizer is ~1.6x-2x better than…
you use the wrong word the Anthropic tokenizer is not worse, its more expensive/verbose
So, worse? Because we benchmark off token use when talking about token use, and everyone else understood that.
Re: The real prices of frontier models
#27Yeah, Anthropic's current tokenizer in Sonnet 5/Opus 4.8/Fable 5 is much worse than OpenAI's. Also, OpenAI has been using their current o200k_base from the day GPT-4o came out over two years ago. Just a few of my own tests: - A ~2000-2002 legacy C++ game codebase at about ~90kloc: GPT 1.12M, Claude 2.2M - A ~30kloc TypeScript codebase: GPT 260K, Claude 437K In the end, GPT's current tokenizer is ~1.6x-2x better than…
Interesting... Naively I'd assume you'd have a pretty unfair advantage on quality if you have materially more information dense tokens. That doesn't really appear to be the case as GPT and Anthropic models appear evenly matched despite Anthropic encoding the same text into almost ~2x the tokens... I'd also - naively - assume this would make training their models more expensive. Though inference now dominates, and the…
more dense tokens means more stuff to fit into the embedding space which is per token, so more work to disentangle later
Re: The real prices of frontier models
#28Re: The real prices of frontier models
#29I have reduced usage of Fable and Sonnet 5 to a minimum. Fable in particular is amazing at creative tasks, but not worth the cost for almost everything else. I can have Opus 4.6/4.7 running non-stop without hitting quota, vs maybe 20 minutes of Fable usage.
Fable can solve the problems Opus couldn't. BUT most of the time I'm not having those kinds of problems. I wouldn't say I'm doing anything groundbreaking but definitely at times obscure and that's when Fable has been able to dig me out of the rut. (the alternative I was actually following was reading textbooks myself to understand the domain better)
The weird twist here is that I've found that there are times where Fable is actually worse than Opus at a straightforward task, not just more expensive. It'll launch off into its own little world for an extended thinking, then start editing files, and I'll already have spent $5 in tokens before I see enough response to know that it's on the wrong track.
Opus's verbosity is actually a boon sometimes for catching false starts early.
Re: The real prices of frontier models
#30Meta: why are the scrollbars hidden / invisible? Awful UX :(