Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Token Counter, now with model comparisons

simonwillison.net

Post reply on HN
Print view View original

61–70 of 93 posts

Re: Claude Token Counter, now with model comparisons

#61

Post by Majromax »

hugodan wrote: post #38

Aren't these increases offset by the quality of the responses and reducing the iterations needed to fine-tune the responses?

Only for the range of tasks where 4.7 performs well but 4.6 performed suboptimally. If both models can one-shot the task without retries, then the number of iterations is already at the lower bound.

This also applies at the sub-task level. If both models need to read three files to figure out which one implements the function they need to modify, then the token tax is paid for all three files even though "not the right file" is presumably an easy conclusion to draw.

This is also related to the challenge of optimizing subagents. Presumably the outer, higher-capacity model can perform better with everything in its context (up to limits), but dispatching a less-capable subagent for a problem might be cheaper overall. Anthropic has a 5:1 cost on input tokens between Opus and Haiku, but Google has 8:1 (Gemini Pro : Flash Lite) and OpenAI has 12:1 (GPT 4.2 : 4.2 nano).

Re: Claude Token Counter, now with model comparisons

#62

Post by dandaka »

Jtarii wrote: post #56
abrookewood wrote: post #24

Earlier quoted context omitted.

Because everyone burns through their limits much faster, forcing them to upgrade to higher limits or new tiers.

I think someone would much sooner switch to a competitor than up their tier.

If model provider believes they have a better model, it can be a viable bet. But many (me included) started experimenting with other providers because of enshittification from Anthropic (price + uptime). Only to find, that Codex is not that worse in quality for a significantly more output per $.

Re: Claude Token Counter, now with model comparisons

#63

Post by Symmetry »

great_psy wrote: post #6

Is there any provided reason from anthropic why they changed the tokenizer ? Is there a quality increase from this change or is it a money grab ?

If they wanted they could always just double the $/token. They don't seem to be able to keep up with their current demand and that's what companies normally do in that circumstance if they're looking to money grab, no need for the bankshot approach.

Re: Claude Token Counter, now with model comparisons

#64

Post by o10449366 »

Esophagus4 wrote: post #60

Anyone have good tips or resources on token management best practices? Because I’ve hit the limiter with one single prompt now on Opus 4.7. What I’m reading so far seems to be: -selective use of models based on task complexity -encoding large repos into more digestible and relevant data structures to reduce constant reingesting -ask Claude to limit output to X tokens (as output tokens are more expensive) -reduce flai…

What was your single prompt? That seems highly unlikely.

Re: Claude Token Counter, now with model comparisons

#65

Post by Esophagus4 »

o10449366 wrote: post #64
Esophagus4 wrote: post #60

Anyone have good tips or resources on token management best practices? Because I’ve hit the limiter with one single prompt now on Opus 4.7. What I’m reading so far seems to be: -selective use of models based on task complexity -encoding large repos into more digestible and relevant data structures to reduce constant reingesting -ask Claude to limit output to X tokens (as output tokens are more expensive) -reduce flai…

What was your single prompt? That seems highly unlikely.

“Ok Claude, I bet you can’t hit the usage limit in one shot… let’s see what you got you little token gremlin”

;)

I just asked it to do a security analysis in a subagent of an unmaintained browser extension and then go fix vulnerabilities it found so I can use it without worrying.

Re: Claude Token Counter, now with model comparisons

#66

Post by slawr1805 »

Esophagus4 wrote: post #60

Anyone have good tips or resources on token management best practices? Because I’ve hit the limiter with one single prompt now on Opus 4.7. What I’m reading so far seems to be: -selective use of models based on task complexity -encoding large repos into more digestible and relevant data structures to reduce constant reingesting -ask Claude to limit output to X tokens (as output tokens are more expensive) -reduce flai…

Thank you for sharing these tips! Just checked RTK and Headroom out and looks like Headroom actually uses RTK under the hood for CLI output compression: https://github.com/chopratejas/headroom#compared-to

Re: Claude Token Counter, now with model comparisons

#67

Post by sumeno »

simianwords wrote: post #46
abrookewood wrote: post #24

Earlier quoted context omitted.

Because everyone burns through their limits much faster, forcing them to upgrade to higher limits or new tiers.

They could just increase the token cost no? There’s little need for cute conspiracies like these

They would have to tell people if they did that.

Re: Claude Token Counter, now with model comparisons

#68

Post by ebcode »

Esophagus4 wrote: post #60

Anyone have good tips or resources on token management best practices? Because I’ve hit the limiter with one single prompt now on Opus 4.7. What I’m reading so far seems to be: -selective use of models based on task complexity -encoding large repos into more digestible and relevant data structures to reduce constant reingesting -ask Claude to limit output to X tokens (as output tokens are more expensive) -reduce flai…

I’m working on a tool that is a more token-efficient code search than grep. I don’t have hard numbers yet, but it’s been working for me to get longer sessions. https://github.com/ebcode/SourceMinder
Post reply on HN