Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Token Counter, now with model comparisons

simonwillison.net

Post reply on HN
Print view View original

81–90 of 93 posts

Re: Claude Token Counter, now with model comparisons

#81

Post by svnt »

vfalbor wrote: post #58
akdor1154 wrote: post #54

Earlier quoted context omitted.

I have often wondered if Chinese is a much 'better' language for LLMs - every character is a token, boom you're done. No weird subword nonsense, no strange semantics being applied to arbitrary chunks of words.. I feel like there must be benefits to being able to have the language tokenized in what must be very close to 1:1.

Yes, it is. In fact, I made a small application to reduce the token consumption for translating from one language to another, and I even invented a language called Tokinensis, which is a mix of different languages, and I ran my own tests with savings of 30%. Chinese is amazing because they encapsulate a ton of information in a single symbol, so you can save a ton of tokens.

Are you able to use the language practically? How would that work? You prompt it in english but tell it to work in tokinensis? And then translate back at the end?

Re: Claude Token Counter, now with model comparisons

#82

Post by vfalbor »

svnt wrote: post #81
vfalbor wrote: post #58

Earlier quoted context omitted.

Yes, it is. In fact, I made a small application to reduce the token consumption for translating from one language to another, and I even invented a language called Tokinensis, which is a mix of different languages, and I ran my own tests with savings of 30%. Chinese is amazing because they encapsulate a ton of information in a single symbol, so you can save a ton of tokens.

Are you able to use the language practically? How would that work? You prompt it in english but tell it to work in tokinensis? And then translate back at the end?

Yep, actually, is a mixture that works. I actually run for my day to day, and I can save tokens, maybe not that I will expected, but it works, you can try if you wish https://translation.tokenstree.com.

Re: Claude Token Counter, now with model comparisons

#83

Post by svnt »

simianwords wrote: post #46
abrookewood wrote: post #24

Earlier quoted context omitted.

Because everyone burns through their limits much faster, forcing them to upgrade to higher limits or new tiers.

They could just increase the token cost no? There’s little need for cute conspiracies like these

There are no conspiracies where a corporation has profit incentive. There is perhaps a question of planning and initial intentionality, but the metrics and motivation to continue are clear enough.

Re: Claude Token Counter, now with model comparisons

#84

Post by vfalbor »

touristtam wrote: post #79
vfalbor wrote: post #58

Earlier quoted context omitted.

Yes, it is. In fact, I made a small application to reduce the token consumption for translating from one language to another, and I even invented a language called Tokinensis, which is a mix of different languages, and I ran my own tests with savings of 30%. Chinese is amazing because they encapsulate a ton of information in a single symbol, so you can save a ton of tokens.

Interested; I came across a post that was mentioning using Kanji for specific use to reduce context.

Maybe in future there will be some "Tokenensis" but in kanjis which could concentrate a lot of info into little space.

Re: Claude Token Counter, now with model comparisons

#85

Post by jmalicki »

akdor1154 wrote: post #54
vfalbor wrote: post #53

This is perfectly legitimate. It's something I've been denouncing day after day. Company X charges you 10dolar per token, while company Y charges you 7dolar, yet company X is cheaper because of the tokenizer they use. The token consumption depends on the tokenizer, and companies create tokenizers using standard algorithms like BPE. But they're charging for hardware access, and the system can be biased to the point th…

I have often wondered if Chinese is a much 'better' language for LLMs - every character is a token, boom you're done. No weird subword nonsense, no strange semantics being applied to arbitrary chunks of words.. I feel like there must be benefits to being able to have the language tokenized in what must be very close to 1:1.

I'm not sure what the state of the art is today, but 15 years ago I worked on a cross-lingual search engine - a challenge with Chinese was that ngram-like models for detecting common language errors (such as typos) were simply ineffective due to this.

We found a lot of gain by having ranking features based on Pinyin to detect typos/misspellings due to homophones (and similar sounding words). I was investigating stroke decomposition to try to be able to detect near homographs, but wasn't able to find any good libraries at the time.

I could imagine the homophone issue is especially relevant for spoken input to LLMs. LLMs are good enough that they're usually right, so it's probably less of an issue, but in English I can have crazy typos and everything just works, I am curious how well that would work for Chinese, since I suspect it's a harder problem by far due to the lack of subword tokens?

Re: Claude Token Counter, now with model comparisons

#86

Post by o10449366 »

Esophagus4 wrote: post #65
o10449366 wrote: post #64

Earlier quoted context omitted.

What was your single prompt? That seems highly unlikely.

“Ok Claude, I bet you can’t hit the usage limit in one shot… let’s see what you got you little token gremlin” ;) I just asked it to do a security analysis in a subagent of an unmaintained browser extension and then go fix vulnerabilities it found so I can use it without worrying.

So it sounds like you were using it on auto mode then if it went ahead and fixed the vulnerabilities without additional turns? If so, that isn't really a single prompt.

Re: Claude Token Counter, now with model comparisons

#87

Post by DANmode »

akdor1154 wrote: post #54
vfalbor wrote: post #53

This is perfectly legitimate. It's something I've been denouncing day after day. Company X charges you 10dolar per token, while company Y charges you 7dolar, yet company X is cheaper because of the tokenizer they use. The token consumption depends on the tokenizer, and companies create tokenizers using standard algorithms like BPE. But they're charging for hardware access, and the system can be biased to the point th…

I have often wondered if Chinese is a much 'better' language for LLMs - every character is a token, boom you're done. No weird subword nonsense, no strange semantics being applied to arbitrary chunks of words.. I feel like there must be benefits to being able to have the language tokenized in what must be very close to 1:1.

If you’re enjoying this thread, you have to make a stop over here:

https://github.com/JuliusBrussee/caveman

Re: Claude Token Counter, now with model comparisons

#88

Post by Esophagus4 »

o10449366 wrote: post #86
Esophagus4 wrote: post #65

Earlier quoted context omitted.

“Ok Claude, I bet you can’t hit the usage limit in one shot… let’s see what you got you little token gremlin” ;) I just asked it to do a security analysis in a subagent of an unmaintained browser extension and then go fix vulnerabilities it found so I can use it without worrying.

So it sounds like you were using it on auto mode then if it went ahead and fixed the vulnerabilities without additional turns? If so, that isn't really a single prompt.

It got limited while it was reading the repo before starting to fix, but yes you’re right. I think I was in auto mode or at least approving all the exploration.

Re: Claude Token Counter, now with model comparisons

#89

Post by tethys »

weird-eye-issue wrote: post #16
mudkipdev wrote: post #4

Why do you need an API key to tokenize the text? Isn't it supposed to be a cheap step that everything else in the model relies on?

To prevent abuse? It's a completely free endpoint so I don't understand your complaint.

It may be free, but it cannot be used without credits.

  Error: {"type":"error","error":{"type":"invalid_request_error","message":"Your credit balance is too low to access the Anthropic API. Please go to Plans & Billing to upgrade or purchase credits."},"request_id":"req_011CaGaBf6uTHfbmdZ39nx1Z"}

Re: Claude Token Counter, now with model comparisons

#90

Post by weird-eye-issue »

tethys wrote: post #89
weird-eye-issue wrote: post #16

Earlier quoted context omitted.

To prevent abuse? It's a completely free endpoint so I don't understand your complaint.

It may be free, but it cannot be used without credits. Error: {"type":"error","error":{"type":"invalid_request_error","message":"Your credit balance is too low to access the Anthropic API. Please go to Plans & Billing to upgrade or purchase credits."},"request_id":"req_011CaGaBf6uTHfbmdZ39nx1Z"}

Again it is to help prevent abuse I don't really see how this is a valid concern? Tokenization is actually fairly CPU intensive
Post reply on HN