The real elephant in the room is pricing for KV cache writes and reads. That makes all the difference for tasks with large context.
The real prices of frontier models
11–20 of 91 posts
Re: The real prices of frontier models
#12The real elephant in the room is pricing for KV cache writes and reads. That makes all the difference for tasks with large context.
[dead]
Re: The real prices of frontier models
#13I have reduced usage of Fable and Sonnet 5 to a minimum. Fable in particular is amazing at creative tasks, but not worth the cost for almost everything else. I can have Opus 4.6/4.7 running non-stop without hitting quota, vs maybe 20 minutes of Fable usage.
Re: The real prices of frontier models
#14Are there any advantages of the new tokenizer? Does it have a larger or smaller vocabulary or just differently weighted?
Well, in my view, it's just the most ordinary manipulation to avoid creating unrest. There is most likely no improvement inside.
Of course, these are my guesses, but did anyone feel the difference in the transition from Opus 4.5 to 4.6? In my opinion, no. And it's unlikely to be a matter of the tokenizer.
Re: The real prices of frontier models
#15I have reduced usage of Fable and Sonnet 5 to a minimum. Fable in particular is amazing at creative tasks, but not worth the cost for almost everything else. I can have Opus 4.6/4.7 running non-stop without hitting quota, vs maybe 20 minutes of Fable usage.
Well, I both agree and disagree with you.
On one hand, the price is just astronomical for Fable, well, not exactly astronomical, but I would say unaffordable. That is to say, so expensive that it is impossible to use.
But on the other hand, Fable is simply incomparable to anything else. I mean, it is just amazing. There is nothing even close to being equal to it.
Re: The real prices of frontier models
#16The fact that OpenAI documents theirs is already a big improvement over Anthropic. But, also, the OpenAI tokenizer got more efficient when they last updated it, rather than less. https://mdstudio.app/o200k-base-tokenizer
[dead]
Re: The real prices of frontier models
#17An individual token, and the level of energy it represents (electricity, or relative effectiveness per model) increasingly seems the space of obsfucation.
This space can be increasingly avoided by becoming, and remaining, efficient and effective with prompts.
Re: The real prices of frontier models
#18I have reduced usage of Fable and Sonnet 5 to a minimum. Fable in particular is amazing at creative tasks, but not worth the cost for almost everything else. I can have Opus 4.6/4.7 running non-stop without hitting quota, vs maybe 20 minutes of Fable usage.
Fable can solve the problems Opus couldn't. BUT most of the time I'm not having those kinds of problems.
I wouldn't say I'm doing anything groundbreaking but definitely at times obscure and that's when Fable has been able to dig me out of the rut. (the alternative I was actually following was reading textbooks myself to understand the domain better)
Re: The real prices of frontier models
#19Yeah, Anthropic's current tokenizer in Sonnet 5/Opus 4.8/Fable 5 is much worse than OpenAI's. Also, OpenAI has been using their current o200k_base from the day GPT-4o came out over two years ago. Just a few of my own tests: - A ~2000-2002 legacy C++ game codebase at about ~90kloc: GPT 1.12M, Claude 2.2M - A ~30kloc TypeScript codebase: GPT 260K, Claude 437K In the end, GPT's current tokenizer is ~1.6x-2x better than…
Interesting... Naively I'd assume you'd have a pretty unfair advantage on quality if you have materially more information dense tokens.
That doesn't really appear to be the case as GPT and Anthropic models appear evenly matched despite Anthropic encoding the same text into almost ~2x the tokens...
I'd also - naively - assume this would make training their models more expensive. Though inference now dominates, and they'd probably rather have more tokens than less (to charge you for them at future 80% margins).
Re: The real prices of frontier models
#20The fact that OpenAI documents theirs is already a big improvement over Anthropic. But, also, the OpenAI tokenizer got more efficient when they last updated it, rather than less. https://mdstudio.app/o200k-base-tokenizer
Interesting. New models are estimated at ~5T params, so 45,000x increase over BERT base (110m). But vocab size of 200k, so only an increase of 7x over BERT base (30k).