It feels like a dedicated orchestration/planning agent needs to be much clearer on costs now as part of the tast plan. Multiple models used at different stages depending on the task.
Measuring Claude 4.7's tokenizer costs
491–500 of 540 posts
Re: Measuring Claude 4.7's tokenizer costs
#492Earlier quoted context omitted.
The cost to hire a human is highly predictable. The cost of AI isn't. I, as a human, need food and shelter, which puts a ceiling to my bargaining power. I can't withdraw my labour indefinitely. The power dynamics are also vastly against me. I represent a fraction of my employer's labour, but my employer represents 100% of my income. That dynamic is totally inverted with AI. You are a rounding error on their revenue s…
By continuously testing competitors and local LLMs? The reason for rising prices is that they (Anthropic) probably realized that they have reached a ceiling of what LLMs are capable of, and while it's a lot, it is still not a big moat and it's definitely not intelligence.
Note that it is very likely this market can't sustain this level of competition for long. We are all still chasing the carrot of AGI, while hardware costs skyrocket.
Re: Measuring Claude 4.7's tokenizer costs
#493Earlier quoted context omitted.
Yes, MarginLab only tests 50 tasks a day, which is too few to give a narrower confidence interval. On the other hand, this really calls into question claims of performance degradation that are based on less intensive use than that. Variance is just so high that long streaks of bad luck are to be expected and plausibly the main source of such complaints. Similarly, it's unlikely you can measure a significant performan…
> Similarly, it's unlikely you can measure a significant performance difference between models like GPT 5.4-xhigh and GPT 5.2 unless you have a task where one of them almost always fails or one almost always succeeds That feels like a concession to the limited benchmarking framework. 5.4-xhigh is supposed to be (and is widely believe to be) a better model than 5.2, so if that's invisible in the benchmarking scores th…
Here's a sample-size calculator that may help illustrate the issue: https://sample-size.net/sample-size-proportions/ Put in the benchmark score of one model as p₀ and of the other model as p₁ (as a fraction between 0 and 1) and observe what kind of sample size you need to reliably observe a significant difference. The largest change between GPT 5.2 and 5.4 highlighted in https://openai.com/index/introducing-gpt-5-4/ is OSWorld-Verified going from 47.3% to to 75.0%. That's quite the difference, right? So plug in 0.473 and 0.75 and note that the required sample size per model is 55. For the software engineering tasks in SWE-Bench Pro, the change from 55.6% to 57.7% is a whopping 2.1 percentage points, which you can detect with a mere 8836 samples.
I'm sure someone in charge of benchmarking at OpenAI knows how statistics work and always makes sure to take a sufficiently large number of samples when comparing different models, but for most other people who want to know which model is better, the answer is unlikely to be worth the cost of measuring it precisely enough to find out.
Re: Measuring Claude 4.7's tokenizer costs
#494The "multiplier" on Github Copilot went from 3 to 7.5. Nice to see that it is actually only 20-30% and Microsoft wanting to lose money slightly slower. https://docs.github.com/fr/copilot/reference/ai-models/suppo...
And that is temporary pricing.
Looking at 4.6 fast, I'm assuming this price will go up to 15 once the promo ends
Re: Measuring Claude 4.7's tokenizer costs
#495Earlier quoted context omitted.
This seems like the experience I've had with every model I've tried over the last several years. It seems like an inherent limitation of the technology, despite the hyperbolic claims of those financially invested in all of this paying off.
Opus 4.6 pre-nerf was incredible, almost magical. It changed my understanding of how good models could be. But that's the only model that ever made me feel that way.
Did they nerf the model or was it changes to Claude code? I agree it got frustrating.
Re: Measuring Claude 4.7's tokenizer costs
#496I did some work yesterday with Opus and found it amazing. Today we are almost on non-speaking terms. I'm asking it to do some simple stuff and he's making incredible stupid mistakes: This is the third time that I have to ask you to remove the issue that was there for more than 20 hours. What is going on here? and at the same time the compacting is firing like crazy. (What adds ~4 minute delays every 1 - 15 minutes) |…
This is not how AI works man. Speaking condescendingly or sternly to it WILL result in worse output. Imagine if you spoke to an intern like that, would they make more or less mistakes after?
You should just revert the context and provide more detail and rationale in the message.
Re: Measuring Claude 4.7's tokenizer costs
#497Earlier quoted context omitted.
Opus 4.6 pre-nerf was incredible, almost magical. It changed my understanding of how good models could be. But that's the only model that ever made me feel that way.
Yes! I genuinely got a LOT of shit done with Opus 4.6 "pre nerf" with regular old out-of-the-box config, no crazy skills or hacks or memory tweaks or anything. The downfall is palpable. Textbook rugpull.
There was no nerf - this meme needs to die.
Re: Measuring Claude 4.7's tokenizer costs
#498Earlier quoted context omitted.
I still haven't seen any statistically sound data supporting that this is happening on the API (per-token pricing.) If you've got something to share I'd love to see it.
There's an interesting analysis here: https://github.com/anthropics/claude-code/issues/42796 >The most striking row is user prompts: 5,608 in February vs 5,701 in March. The human put in the same effort. But the model consumed 80x more API requests and 64x more output tokens to produce demonstrably worse results.
[deleted]
Re: Measuring Claude 4.7's tokenizer costs
#499Earlier quoted context omitted.
The docs suggest not using max effort in most cases to avoid overthinking :shrug:
They've jumped the shark. I truly can't comprehend why all of these changes were necessary. They had a literal money printing machine that actually got real shit done, really well. Now it's a gamble every time and I am pulling back hard from Anthropic ecosystem.
it's clearly all in your head. 4.6 is just as capable as it used to be. literally no one on the internet has managed to post credulous and real evidence of a nerf
this is just another trendy conspiracy theory that people reinforce because of selection/recency bias. you hear "nerf", your brain overindexes on the next time Claude does poorly. it is the same phenomenon when you notice a new vocabulary word all the time.
Re: Measuring Claude 4.7's tokenizer costs
#500We noticed this two weeks ago where we found some of our requests are unexpected took more tokens than measured by count_tokens call. At the end they were Anthropic's A/B testing routing some Opus 4.6 calls to Opus 4.7. https://matrix.dev/blog-2026-04-16.html (We were talking to Opus 4.7 twelve days ago)
Wonder what they do for their token cache if they swap mid-session like that.