Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Token Counter, now with model comparisons

simonwillison.net

Post reply on HN
Print view View original

71–80 of 93 posts

Re: Claude Token Counter, now with model comparisons

#71

Post by Esophagus4 »

ebcode wrote: post #68
Esophagus4 wrote: post #60

Anyone have good tips or resources on token management best practices? Because I’ve hit the limiter with one single prompt now on Opus 4.7. What I’m reading so far seems to be: -selective use of models based on task complexity -encoding large repos into more digestible and relevant data structures to reduce constant reingesting -ask Claude to limit output to X tokens (as output tokens are more expensive) -reduce flai…

I’m working on a tool that is a more token-efficient code search than grep. I don’t have hard numbers yet, but it’s been working for me to get longer sessions. https://github.com/ebcode/SourceMinder

Oh nice! Thank you, I will definitely give this a shot.

I was looking at tree-sitter myself for this task.

Re: Claude Token Counter, now with model comparisons

#72

Post by Esophagus4 »

slawr1805 wrote: post #66
Esophagus4 wrote: post #60

Anyone have good tips or resources on token management best practices? Because I’ve hit the limiter with one single prompt now on Opus 4.7. What I’m reading so far seems to be: -selective use of models based on task complexity -encoding large repos into more digestible and relevant data structures to reduce constant reingesting -ask Claude to limit output to X tokens (as output tokens are more expensive) -reduce flai…

Thank you for sharing these tips! Just checked RTK and Headroom out and looks like Headroom actually uses RTK under the hood for CLI output compression: https://github.com/chopratejas/headroom#compared-to

Didn’t realize that, thank you!

Re: Claude Token Counter, now with model comparisons

#73

Post by aftbit »

stingraycharles wrote: post #47
kouteiheika wrote: post #8

Earlier quoted context omitted.

I'd guess it's because they don't want people to reverse engineer it. Note that they're the only provider which doesn't make their tokenizer available offline as a library (i.e. the only provider whose tokenizer is secret).

Anthropic is somewhat becoming the Apple of AI in terms of closed ecosystem. Not saying I blame them, I just don't like it as a customer. The fact that it's impossible to get the actual thinking tokens anymore, but we have to do with a rewritten summary, is extremely off-putting. I understand that it's necessary for users, but when writing agentic applications yourself, it's super annoying not to have the actual reas…

It's _not_ that it's necessary for users. It's that Anthropic got Opus 4.6 ripped off so hard by MiniMax that they no longer want to expose true thinking tokens to random developers. If you're one of the blessed class, you can still get real thinking tokens, but you need to be a major enterprise customer, like the companies that they gave Mythos access.

Re: Claude Token Counter, now with model comparisons

#74

Post by ebcode »

Esophagus4 wrote: post #71
ebcode wrote: post #68

Earlier quoted context omitted.

I’m working on a tool that is a more token-efficient code search than grep. I don’t have hard numbers yet, but it’s been working for me to get longer sessions. https://github.com/ebcode/SourceMinder

Oh nice! Thank you, I will definitely give this a shot. I was looking at tree-sitter myself for this task.

It's still in beta, and I'm hoping to get more feedback, so feel free to post in the issues or reach out directly if you run into any problems.

Re: Claude Token Counter, now with model comparisons

#75

Post by hyperpape »

This is a great piece of data, but only a piece of the actual question that we need to answer, which is:

For a given input, how many tokens will be used for an answer, and how high quality will that answer be?

Measuring the tokenizer is just one input into the cost-benefit tradeoff.

Re: Claude Token Counter, now with model comparisons

#77

Post by sigmoid10 »

amelius wrote: post #57
sigmoid10 wrote: post #23

Earlier quoted context omitted.

I never understood why people want this in the first place. Sure, making this step more human explainable would be nice and possibly even fix some very particular problems for particular languages, but it directly goes against the primary objective of a tokenizer: Optimizing sequence length vs. vocabulary size. This is a pretty clear and hard optimization target and the best you can do is make sure that your tokenize…

If you want to make it more human-explainable, then ditch the entire tokenizer and just feed the models raw characters. Because now there is nothing to explain.

Then that means you need at least 4x the compute to achieve the same results as state of the art. Meaning that if I can train my frontier model with my normal tokenizer in 3 months, it will take you a year. When major releases across all competing providers are measured in months, there's simply no incentive to do that just to capture these fringe edge cases.

Re: Claude Token Counter, now with model comparisons

#78

Post by ctoth »

kouteiheika wrote: post #7

> Opus 4.7 tokenizer used 1.46x the number of tokens as Opus 4.6 Interesting. Unfortunately Anthropic doesn't actually share their tokenizer, but my educated guess is that they might have made the tokenizer more semantically aware to make the model perform better. What do I mean by that? Let me give you an example. (This isn't necessarily what they did exactly; just illustrating the idea.) Let's take the gpt-oss-120b…

Poor Ed.

Re: Claude Token Counter, now with model comparisons

#79

Post by touristtam »

vfalbor wrote: post #58
akdor1154 wrote: post #54

Earlier quoted context omitted.

I have often wondered if Chinese is a much 'better' language for LLMs - every character is a token, boom you're done. No weird subword nonsense, no strange semantics being applied to arbitrary chunks of words.. I feel like there must be benefits to being able to have the language tokenized in what must be very close to 1:1.

Yes, it is. In fact, I made a small application to reduce the token consumption for translating from one language to another, and I even invented a language called Tokinensis, which is a mix of different languages, and I ran my own tests with savings of 30%. Chinese is amazing because they encapsulate a ton of information in a single symbol, so you can save a ton of tokens.

Interested; I came across a post that was mentioning using Kanji for specific use to reduce context.

Re: Claude Token Counter, now with model comparisons

#80

Post by amelius »

sigmoid10 wrote: post #77
amelius wrote: post #57

Earlier quoted context omitted.

If you want to make it more human-explainable, then ditch the entire tokenizer and just feed the models raw characters. Because now there is nothing to explain.

Then that means you need at least 4x the compute to achieve the same results as state of the art. Meaning that if I can train my frontier model with my normal tokenizer in 3 months, it will take you a year. When major releases across all competing providers are measured in months, there's simply no incentive to do that just to capture these fringe edge cases.

Yes, OK. But all the tutorials start with explaining how a tokenizer works. This is not necessary. And in fact makes the message of why a tokenizer is necessary not come across as well.
Post reply on HN