Aren't these increases offset by the quality of the responses and reducing the iterations needed to fine-tune the responses?
Likely only some of the time
Claude Token Counter, now with model comparisons
51–60 of 93 posts
Re: Claude Token Counter, now with model comparisons
#52[flagged]
Re: Claude Token Counter, now with model comparisons
#53This is perfectly legitimate. It's something I've been denouncing day after day. Company X charges you 10dolar per token, while company Y charges you 7dolar, yet company X is cheaper because of the tokenizer they use. The token consumption depends on the tokenizer, and companies create tokenizers using standard algorithms like BPE. But they're charging for hardware access, and the system can be biased to the point that if you speak in English, you consume 17% less than if your prompt is written in Spanish, or even if you write with Chinese characters, you'll significantly reduce your token consumption compared to English speakers. I've written about this several times on HN, but for whatever reason, every time I mention it, they flag my post.
Re: Claude Token Counter, now with model comparisons
#54This is perfectly legitimate. It's something I've been denouncing day after day. Company X charges you 10dolar per token, while company Y charges you 7dolar, yet company X is cheaper because of the tokenizer they use. The token consumption depends on the tokenizer, and companies create tokenizers using standard algorithms like BPE. But they're charging for hardware access, and the system can be biased to the point th…
I have often wondered if Chinese is a much 'better' language for LLMs - every character is a token, boom you're done. No weird subword nonsense, no strange semantics being applied to arbitrary chunks of words.. I feel like there must be benefits to being able to have the language tokenized in what must be very close to 1:1.
Re: Claude Token Counter, now with model comparisons
#55This is the rugpull that is starting to push me to reconsider my use of Claude subscriptions. The "free ride" part of this being funded as a loss leader is coming to a close. While we break away from Claude, my hope is that I can continue to send simple problems to very smart local llms (qwen 3.6, I see you) and reserve Claude for purely extreme problems appropriate for it's extreme price.
Quality of answers from quantized models is noticeable worse than using the full model. You'll be better using Qwen 3.6 Plus through Alibaba coding plan.
> Quality of answers from quantized models is noticeable worse than using the full model.
This is the very reason I've heard I shouldn't use Alibaba!
Re: Claude Token Counter, now with model comparisons
#56Earlier quoted context omitted.
How would it be a money grab? If the new tokenizer requires more tokens to encode the same information, it costs them more money for inference. The point of charging per token is that the cost is proportional to the number of tokens. That's my understanding anyway
Because everyone burns through their limits much faster, forcing them to upgrade to higher limits or new tiers.
I think someone would much sooner switch to a competitor than up their tier.
Re: Claude Token Counter, now with model comparisons
#57Earlier quoted context omitted.
There is currently very little evidence that morphological tokenizers help model performance [1]. For languages like German (where words get glued together) there is a bit more evidence (eg a paper I worked on [2]), but overall I start to suspect the bitter lesson is also true for tokenization. [1] https://arxiv.org/pdf/2507.06378 [2] https://pieter.ai/bpe-knockout/
I never understood why people want this in the first place. Sure, making this step more human explainable would be nice and possibly even fix some very particular problems for particular languages, but it directly goes against the primary objective of a tokenizer: Optimizing sequence length vs. vocabulary size. This is a pretty clear and hard optimization target and the best you can do is make sure that your tokenize…
If you want to make it more human-explainable, then ditch the entire tokenizer and just feed the models raw characters. Because now there is nothing to explain.
Re: Claude Token Counter, now with model comparisons
#58This is perfectly legitimate. It's something I've been denouncing day after day. Company X charges you 10dolar per token, while company Y charges you 7dolar, yet company X is cheaper because of the tokenizer they use. The token consumption depends on the tokenizer, and companies create tokenizers using standard algorithms like BPE. But they're charging for hardware access, and the system can be biased to the point th…
I have often wondered if Chinese is a much 'better' language for LLMs - every character is a token, boom you're done. No weird subword nonsense, no strange semantics being applied to arbitrary chunks of words.. I feel like there must be benefits to being able to have the language tokenized in what must be very close to 1:1.
Yes, it is. In fact, I made a small application to reduce the token consumption for translating from one language to another, and I even invented a language called Tokinensis, which is a mix of different languages, and I ran my own tests with savings of 30%. Chinese is amazing because they encapsulate a ton of information in a single symbol, so you can save a ton of tokens.
Re: Claude Token Counter, now with model comparisons
#59Earlier quoted context omitted.
> This is the rugpull that is starting to push me to reconsider my use of Claude subscriptions. I'm still with them cause the model is good, but yes, I'm noticing my limits burning up somewhat faster on the 100 USD tier, I bet the 20 USD tier is even more useless. I wouldn't call it a rugpull, since it seems like there might be good technical reasons for the change, but at the same time we won't know for sure if they…
20 USD tier was useless from the start. You'd get to the limit in 30 minutes. Codex with 20 USD on the other hand...
Give Codex a month or two and it'll just do the same thing, though now with a million more users because Claude Code wasn't good enough.
Re: Claude Token Counter, now with model comparisons
#60Anyone have good tips or resources on token management best practices? Because I’ve hit the limiter with one single prompt now on Opus 4.7.
What I’m reading so far seems to be:
-selective use of models based on task complexity
-encoding large repos into more digestible and relevant data structures to reduce constant reingesting
-ask Claude to limit output to X tokens (as output tokens are more expensive)
-reduce flailing by giving plenty of input context
-use Headroom and RTK
-disable unused MCP, move stuff from CLAUDE.md to skills
But I’d love to learn if anyone has any good tips, links, or tools as I’m getting rate limited twice a day now.