Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Which programming languages are most token-efficient?

martinalderson.com

Post reply on HN
Print view View original

91–96 of 96 posts

Re: Which programming languages are most token-efficient?

#91

Post by eigenspace »

switchbak wrote: post #7

I would expect that we’ll end up compressing (or whatever term you would use) this at some point so many of those syntactical differences will not be as significant. But I would love for more expressive and compact languages to do better, selfish as I am. But I think training data size is more of a factor, and we won’t be all moving up Clojure any time soon.

I can't speak to Clojure, but I will say that LLMs are actually surprisingly good at writing and understanding Julia code compared to some languages that have a much larger training corpus to pull from.

Re: Which programming languages are most token-efficient?

#93

Post by muyuu »

eru wrote: post #24
muyuu wrote: post #20

Earlier quoted context omitted.

You could, but you wouldn't when those keywords can all change in equivalent contexts.

What do you mean? `public` might have a token by itself, even though you can have `pub` occurring in other contexts, too.

I meant that it wouldn't be efficient to agglomerate tokens in that way and that's why the system won't do it

Re: Which programming languages are most token-efficient?

#94

Post by muyuu »

janalsncm wrote: post #86
muyuu wrote: post #20

Earlier quoted context omitted.

You could, but you wouldn't when those keywords can all change in equivalent contexts.

The BPE or wordpiece tokenization algorithm will greedily take the longest valid token prefix. So if your text starts with “public static void main” it will try to find the longest token which matches that prefix. Even if “public” is a token, it will prefer to tokenize “public static” together.

yes, but then you have both alternatives as tokens, which nullifies GP's argument

Re: Which programming languages are most token-efficient?

#96

Post by make3 »

gpm wrote: post #34
make3 wrote: post #33

Earlier quoted context omitted.

With Chain of Thoughts (text thinking), the models can already use as much compute as they want in any language (determined by reinforcement learning training)

I'm not convinced that thinking tokens - which sort of have to serve a specific chain of thought purpose - are interchangeable with input tokens during which give the model compute without having it add new text. For a very imperfect human analogy, it feels like saying "a student can spend as much time thinking about the text as they want, so the textbook can be extremely terse". Definitely just gut feelings though -…

We could definitely use RL to add blank, invisible "" tokens whenever the model thinks it should. Or just allow it to say "Hmm." lol.

Easy to test from a technical perspective is all I'm saying, and not a bad idea.

Post reply on HN