Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Token Counter, now with model comparisons

simonwillison.net

Post reply on HN
Print view View original

1–10 of 93 posts

Re: Claude Token Counter, now with model comparisons

#3

Post by simonw »

tomglynch wrote: post #2

Interesting findings. Might need a way to downsample images on upload to keep costs down.

Yeah that should work - it looks like the same pixel dimension image at smaller sizes has about the same token cost for 4.6 and 4.7, so the image cost increase only kicks in if you use larger images that 4.6 would have presumably resized before inspecting.

Re: Claude Token Counter, now with model comparisons

#7

Post by kouteiheika »

> Opus 4.7 tokenizer used 1.46x the number of tokens as Opus 4.6

Interesting. Unfortunately Anthropic doesn't actually share their tokenizer, but my educated guess is that they might have made the tokenizer more semantically aware to make the model perform better. What do I mean by that? Let me give you an example. (This isn't necessarily what they did exactly; just illustrating the idea.)

Let's take the gpt-oss-120b tokenizer as an example. Here's how a few pieces of text tokenize (I use "|" here to separate tokens):

    Kill -> [70074]
    Killed -> [192794]
    kill -> [25752]
    k|illed -> [74, 7905]
    kill -> [15874]
    killed -> [17372]
You have 3 different tokens which encode the same word (Kill, kill, kill) depending on its capitalization and whether there's a space before it or not, you have separate tokens if it's the past tense, etc.

This is not necessarily an ideal way of encoding text, because the model must learn by brute force that these tokens are, indeed, related. Now, imagine if you'd encode these like this:

   |kill
   |kill|ed
   kill|
   kill|ed
   |kill
   |kill|ed
Notice that this makes much more sense now - the model now only has to learn what "" is, what "kill" is, what "" is, and what "ed" (the past tense suffix) is, and it can compose those together. The downside is that it increases the token usage.

So I wouldn't be surprised if this is what they did. Or, my guess number #2, they removed the tokenizer altogether and replaced them with a small trained model (something like the Byte Latent Transformer) and simply "emulate" the token counts.

Re: Claude Token Counter, now with model comparisons

#8

Post by kouteiheika »

mudkipdev wrote: post #4

Why do you need an API key to tokenize the text? Isn't it supposed to be a cheap step that everything else in the model relies on?

I'd guess it's because they don't want people to reverse engineer it.

Note that they're the only provider which doesn't make their tokenizer available offline as a library (i.e. the only provider whose tokenizer is secret).

Re: Claude Token Counter, now with model comparisons

#10

Post by Aurornis »

great_psy wrote: post #6

Is there any provided reason from anthropic why they changed the tokenizer ? Is there a quality increase from this change or is it a money grab ?

The tokenizer is an important part of overall model training and performance. It’s only one piece of the overall cost per request. If a tokenizer that produces more tokens also leads to a model that gets to the correct answer more quickly and requires fewer re-prompts because it didn’t give the right answer, the overall cost can still be lower.

Comparisons are still ongoing but I have already seen some that suggest that Opus 4.7 might on average arrive at the answer with fewer tokens spent, even with the additional tokenizer overhead.

So, no, not a money grab.

Post reply on HN