Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Token Counter, now with model comparisons

simonwillison.net

Post reply on HN
Print view View original

31–40 of 93 posts

Re: Claude Token Counter, now with model comparisons

#31

Post by rimliu »

KronisLV wrote: post #21
aliljet wrote: post #13

This is the rugpull that is starting to push me to reconsider my use of Claude subscriptions. The "free ride" part of this being funded as a loss leader is coming to a close. While we break away from Claude, my hope is that I can continue to send simple problems to very smart local llms (qwen 3.6, I see you) and reserve Claude for purely extreme problems appropriate for it's extreme price.

> This is the rugpull that is starting to push me to reconsider my use of Claude subscriptions. I'm still with them cause the model is good, but yes, I'm noticing my limits burning up somewhat faster on the 100 USD tier, I bet the 20 USD tier is even more useless. I wouldn't call it a rugpull, since it seems like there might be good technical reasons for the change, but at the same time we won't know for sure if they…

20 USD tier was useless from the start. You'd get to the limit in 30 minutes. Codex with 20 USD on the other hand...

Re: Claude Token Counter, now with model comparisons

#32

Post by nl »

kouteiheika wrote: post #28
nl wrote: post #25

Earlier quoted context omitted.

This is almost certainly wrong. Case sensitive language models have been a thing since way before neural language models. I was using them with boosted tree models at least ten years ago, and even my Java NLP tool did this twenty years ago (damn!). There is no novelty there of course - I based that on PG's "A Plan for Spam". See for example CountVectorizer: https://scikit-learn.org/stable/modules/generated/sklearn.fe…

> This is almost certainly wrong. So how would you explain the increase in token usage, considering the fact that conventionally tokenizers are trained to minimize the token usage within a given vocabulary budget? > Putting an inductive bias in your tokenizer seems just a terrible idea. You're already effectively doing this by the sheer fact of using a BPE tokenizer, and especially with modern BPE-based LLM tokenizer…

> So how would you explain the increase in token usage, considering the fact that conventionally tokenizers are trained to minimize the token usage within a given vocabulary budget?

Just modeling whitespace as its own token would seem to explain the increase.

> Qwen3.6 tokenizer: "pretokenizer"

That's the pre-tokenizer, not the tokenizer. That is mostly a performance optimization that lets the memory requirements for the BPE tokenizer be a lot less.

> I could see a world where you could build a better tokenizer training algorithm which would be able to better take the natural morphology of the underlying text into account.

The reason everyone went to BPE was because it was so dramatically better than morphology based tokenizers. See the BPE paper: https://arxiv.org/abs/1508.07909

BPE already learns morphology because it sees the raw bytes.

Re: Claude Token Counter, now with model comparisons

#34

Post by rrr_oh_man »

sigmoid10 wrote: post #23
ipieter wrote: post #19

Earlier quoted context omitted.

There is currently very little evidence that morphological tokenizers help model performance [1]. For languages like German (where words get glued together) there is a bit more evidence (eg a paper I worked on [2]), but overall I start to suspect the bitter lesson is also true for tokenization. [1] https://arxiv.org/pdf/2507.06378 [2] https://pieter.ai/bpe-knockout/

I never understood why people want this in the first place. Sure, making this step more human explainable would be nice and possibly even fix some very particular problems for particular languages, but it directly goes against the primary objective of a tokenizer: Optimizing sequence length vs. vocabulary size. This is a pretty clear and hard optimization target and the best you can do is make sure that your tokenize…

I feel it's a case of "This random word generator can't possibly be smarter than I?!"

Re: Claude Token Counter, now with model comparisons

#35

Post by Grp1 »

rimliu wrote: post #31
KronisLV wrote: post #21

Earlier quoted context omitted.

> This is the rugpull that is starting to push me to reconsider my use of Claude subscriptions. I'm still with them cause the model is good, but yes, I'm noticing my limits burning up somewhat faster on the 100 USD tier, I bet the 20 USD tier is even more useless. I wouldn't call it a rugpull, since it seems like there might be good technical reasons for the change, but at the same time we won't know for sure if they…

20 USD tier was useless from the start. You'd get to the limit in 30 minutes. Codex with 20 USD on the other hand...

OpenAI has been doing the same thing gradually. Codex launched with generous Plus limits, then they introduced the $100 Pro tier, and Plus limits have quietly tightened since. With the same repetitive tasks I was running, consumption is noticeably higher now for the same output.

The pattern feels deliberate — make the $20 tier just uncomfortable enough that power users upgrade, without officially announcing the reduction. If it continues, $20 buys you a demo and $100 buys you actual work.

Re: Claude Token Counter, now with model comparisons

#36

Post by lifis »

I'm really surprised that:

1. Anthropic has not published anything about why they made the change and how exactly they changed it

2. Nobody has reverse engineered it. It seems easy to do so using the free token counting APIs (the Google Vertex AI token count endpoint seems to support 2000 req/min = ~3million req/day, seems enough to reverse engineer it)

Re: Claude Token Counter, now with model comparisons

#37

Post by kouteiheika »

nl wrote: post #32
kouteiheika wrote: post #28

Earlier quoted context omitted.

> This is almost certainly wrong. So how would you explain the increase in token usage, considering the fact that conventionally tokenizers are trained to minimize the token usage within a given vocabulary budget? > Putting an inductive bias in your tokenizer seems just a terrible idea. You're already effectively doing this by the sheer fact of using a BPE tokenizer, and especially with modern BPE-based LLM tokenizer…

> So how would you explain the increase in token usage, considering the fact that conventionally tokenizers are trained to minimize the token usage within a given vocabulary budget? Just modeling whitespace as its own token would seem to explain the increase. > Qwen3.6 tokenizer: "pretokenizer" That's the pre -tokenizer, not the tokenizer. That is mostly a performance optimization that lets the memory requirements fo…

> That's the pre-tokenizer, not the tokenizer.

Yes, it's an extra tokenizer which runs before the learned tokenizer and injects an inductive bias into it.

> That is mostly a performance optimization that lets the memory requirements for the BPE tokenizer be a lot less.

While it does indeed speed up training of the tokenizer, no, it isn't mostly just a performance optimization? It injects a clear cut inductive bias into the tokenizer (split by words, split by punctuation, don't merge words and numbers, etc. -- is that not an inductive bias?), and for some languages (e.g. Asian languages which don't use spaces) the "it's just for performance" argument doesn't make as much sense because there it has no spaces to split on, so the chunks of text are much longer (although it does still split on punctuation, etc.).

Can we not agree that the absolutist position of "Putting an inductive bias in your tokenizer seems just a terrible idea." (as in - any inductive bias) is not actually true, especially since people are actually doing it?

Note, I'm not actually arguing that hand-crafted morphological tokenizers are better. (Which is the straw man many people seem to be replying to.) I'm just arguing that it should be feasible to train your tokenizer in a more morphologically aware way, because BPE doesn't do that.

> The reason everyone went to BPE was because it was so dramatically better than morphology based tokenizers. [..] BPE already learns morphology because it sees the raw bytes.

The reason everyone went to BPE is because of the bitter lesson (and because you don't have to hardcode your whole vocabulary, i.e. no UNK tokens), and not because it's particularly good at learning the morphology of the actual text. It's trivial to show countless examples where it fails to do so.

Re: Claude Token Counter, now with model comparisons

#39

Post by orbital-decay »

friendzis wrote: post #18
kouteiheika wrote: post #7

> Opus 4.7 tokenizer used 1.46x the number of tokens as Opus 4.6 Interesting. Unfortunately Anthropic doesn't actually share their tokenizer, but my educated guess is that they might have made the tokenizer more semantically aware to make the model perform better. What do I mean by that? Let me give you an example. (This isn't necessarily what they did exactly; just illustrating the idea.) Let's take the gpt-oss-120b…

This is such a superficial, English-centric take, but it might as well be true. It seems to me that in non-english languages the models, especially chatgpt, have suffered in the declension department and output words in cases that do not fit the context. I have just ran an experiment: I have taken a word and asked models (chatgpt, gemini and claude) to explode it into parts. The caveat is that it could either be root…

Current models understand different tokenization variants perfectly, e.g. leading space vs no leading space vs one character per token. It doesn't even affect evals and behchmarks. They're also good at languages that have very flexible word formation (e.g. Slavic) and can easily invent pretty natural non-existent words without being restricted by tokenization. This ability took a bit of a hit with recent RL and code generation optimizations, but this is not related to tokenization.

>None of them realized this duality and have taken one possible interpretation.

I suspect this happens due to mode collapse and has nothing to do with the tokenization. Try this with a base model.

Re: Claude Token Counter, now with model comparisons

#40

Post by frb »

kouteiheika wrote: post #7

> Opus 4.7 tokenizer used 1.46x the number of tokens as Opus 4.6 Interesting. Unfortunately Anthropic doesn't actually share their tokenizer, but my educated guess is that they might have made the tokenizer more semantically aware to make the model perform better. What do I mean by that? Let me give you an example. (This isn't necessarily what they did exactly; just illustrating the idea.) Let's take the gpt-oss-120b…

I was looking into morpheme tokenization approach, but went even more radical with building a semantic primitive tokenizer [1], i.e. kill, killed, killer would all share the same semantic connection and tokens, e.g. [KILL], [KILL, BEFORE], [KILL, SOMEONE].

It’s based on semantic primitives (Wierzbicka NSM) and emoji (the fun idea that got me interested in this in the first place).

So far I’ve tested 6 iterations and it trains and responds well with a 10k vocab, but the grammar came out rougher. Working on 8th iteration, mainly to improve the grammar and language. Turns out the smaller vocab couldn’t be maintained and all improvements get us back in the ballpark of the 32k vocab size. Further testing is still outstanding for this week.

[1] https://github.com/frane/primoji

Post reply on HN