11–20 of 96 posts
So I'm not convinced this is either the right metric, or even if you got the right metric that it's a metric you want to minimize.
It strikes me that more tokens likely give the LLM more time/space to "think". Also that more redundant tokens, like local type declarations instead of type inference from far away, likely often reduce the portion of the code LLMs (and humans) have to read. So I'm not convinced this is either the right metric, or even if you got the right metric that it's a metric you want to minimize.
Because that’s what happened in the real world when generating a bunch of untyped Python code.
I am not sure token efficiency is an interesting problem in the long term, though.
And in the short term I wonder if prompts could be pre-compiled to “compressed tokens”; the idea would be to use a smaller number of tokens to represent a frequently needed concept; kind of like LZ compression. Or maybe token compression becomes a feature of future models optimized for specific tasks.
I was wondering last year if it would be worthwhile trying to create a language that was especially LLM-friendly, eg that embedded more context in the language structure. The idea is to make more of the program and the thinking behind it, explicit to the LLM but in a programming language style to eliminate the ambiguity of natural language (one could just use comments).
Then it occurred to me that with current LLM training methodology that there’s a chicken-and-egg problem; it doesn’t start to show rewards until there is a critical mass of good code in the language for LLMs to train on.
Not surprisingly, it is J [1], an APL dialect. [1] https://www.jsoftware.com/
Realistically, it’s also a function of how many iterations it takes for an AI agent to correctly solve a problem with a given language. I’d imagine most AI agents would frequently have to redo J or F# code, as they are fairly uncommon languages with much smaller training set than JavaScript or Python.
I doubt this to be a meaningful metric for anything but code exploration in a larger codebase. E.g. when it comes to authoring code, C, which comes language, is by far one of the languages that LLMs excel most at.
But had never considered that a programming language might be created thats less human readable/auditable to enable LLMs.
Scares me a bit.
This is kind of just a measurement of how representative a language is in the distribution of the tokenizer training. You could have a single token equal to “public static void main”.