Someone has made a programming language called Sui, which is said to be designed for LLMs.
However, using index-based variable names in order to "avoid typo bugs" makes it more difficult than general-purpose languages, and it also has poor token efficiency :(
Which programming languages are most token-efficient?
41–50 of 96 posts
Re: Which programming languages are most token-efficient?
#42I have always had concerns about physical robots making my work less safe in the real world. But had never considered that a programming language might be created thats less human readable/auditable to enable LLMs. Scares me a bit.
LLMs in their current form rely heavily on the vast amount of human data that's available, to learn from it as a first step (the second step is RL). We're not building a language for LLMs just yet.
> We're not building a language for LLMs just yet.
Working on it, actually! I think it's a really interesting problem space - being efficient on tokens, readable by humans for review, strongly typed and static for reasoning purposes, and having extremely regular syntax. One of the biggest issues with symbols is that, to a human, matching parentheses is relatively easy, but the models struggle with it.
I expect a language like the one I'm playing with will mature enough over the next couple years that models with a knowledge cutoff around 1/2027 will probably know how to program it well enough for it to start being more viable.
One of the things I plan to do is build evals so that I can validate the performance of various models on my as yet only partially baked language. I'm also using only LLMs to build out the entire infrastructure, mostly to see if it's possible.
Re: Which programming languages are most token-efficient?
#43An agent can make summaries via Markdown files while processing. Then use that to break the problem to several issues and then tackle them one by one, even automatically, but more usually interactively. The problem is the technique now, not the llm. Yes, it costs a lot (lot) more. But, it can do it, and people work cost way more than tokens.
Re: Which programming languages are most token-efficient?
#44If language supports comments and LLM is allowed to write them (or docstrings, or any such), there go your tokens.
Plus, they will strongly "pull" the context when LLM parses it back, to the point of overriding your instructions (true story)
Re: Which programming languages are most token-efficient?
#45Earlier quoted context omitted.
I've also had decent experiences with Rust recently. I haven't done enough Haskell programming in the AI era to really say. But it could be that different programming languages are a bit like different human languages for these models: when they have more than some threshold of training data, they can express their general problem solving skills in any of them? And then it's down to how much the compiler and linters…
> But it could be that different programming languages are a bit like different human languages for these models: when they have more than some threshold of training data, they can express their general problem solving skills in any of them? And then it's down to how much the compiler and linters can yell at them. I think this is exactly right.
Exactly my opinion - I think the more you lock down the "search space" by providing strong and opinionated tooling, the more LLMs perform well. I think of it as starting something like a simulated annealing trying to get a correct solution, versus the same simulated annealing run while using heuristics and bounds to narrow the solution space
Re: Which programming languages are most token-efficient?
#46Earlier quoted context omitted.
LLMs in their current form rely heavily on the vast amount of human data that's available, to learn from it as a first step (the second step is RL). We're not building a language for LLMs just yet.
> We're not building a language for LLMs just yet. Working on it, actually! I think it's a really interesting problem space - being efficient on tokens, readable by humans for review, strongly typed and static for reasoning purposes, and having extremely regular syntax. One of the biggest issues with symbols is that, to a human, matching parentheses is relatively easy, but the models struggle with it. I expect a lang…
do you expect the model to train on synthetic data or do you expect to grow a userbase that will generate organic training data?
> One of the biggest issues with symbols is that, to a human, matching parentheses is relatively easy, but the models struggle with it.
Great point. I find it near trivial to close parens but llms seem to struggle with the lisps ive played with because of this counting issue. To the point where ive not been working with them as much. typescript and functional js as other commentors note is usually smooth sailing.
Re: Which programming languages are most token-efficient?
#47This is kind of just a measurement of how representative a language is in the distribution of the tokenizer training. You could have a single token equal to “public static void main”.
If you look at the list, you'll see that you're incorrect, as C and JavaScript are not at the top. Seeing all the C languages and JavaScript at the bottom like this makes me wonder if it's not just that Curly brackets take a lot of tokens.
I imagine that having to write
for (int index = 0; index for index in 0...size
Re: Which programming languages are most token-efficient?
#48Earlier quoted context omitted.
> We're not building a language for LLMs just yet. Working on it, actually! I think it's a really interesting problem space - being efficient on tokens, readable by humans for review, strongly typed and static for reasoning purposes, and having extremely regular syntax. One of the biggest issues with symbols is that, to a human, matching parentheses is relatively easy, but the models struggle with it. I expect a lang…
do you expect the model to train on synthetic data or do you expect to grow a userbase that will generate organic training data? > One of the biggest issues with symbols is that, to a human, matching parentheses is relatively easy, but the models struggle with it. Great point. I find it near trivial to close parens but llms seem to struggle with the lisps ive played with because of this counting issue. To the point w…
> do you expect the model to train on synthetic data or do you expect to grow a userbase that will generate organic training data?
Both, essentially, I expect the code examples to grow organically but I expect most of them to come from LLMs, after all, that's the point of the language. I basically expect there to be a step function in effectiveness when the language has been ingested by the models, but they're already plenty decent-ish right now at it.
The most fascinating thing to me, generating the whole thing, has been that the LLMs are really, really good at iterating in a tight loop by updating the interpreter with new syntax, updating the stdlib with that new syntax, building some small extension to try using it, and then surfacing the need for a new builtin or primitive to start the cycle over.
I'm also leaning heavily on Chatgpt-5.2's insanely good math skills, and the language I'm building is very math heavy - it's essentially a distant cousin to Idris or any of the other dependently-typed theorem proving languages.
Re: Which programming languages are most token-efficient?
#4905a1be
Given that APL was punished for using a non-ASCII character set, this would presumably also affect 05AB1E.
Re: Which programming languages are most token-efficient?
#50I'm biased by my preferred style of programming languages but I think that pure statically typed functional languages are incredibly well suited for LLMs. The purity gives you referential transparency and static analysis powers that the LLM can leverage to stay correctly on task. The high level declarative nature and type driven development style of languages like Haskell also make it really easy for an experienced d…
I'm inclined to agree with you in principle, but there's much, much less Haskell examples in their training corpus than for JavaScript or Python.
For the little Haskell I've done with llms, I can tell you they're not bad at it.
Actually, Haskell was a bit too hard for me on my own for real projects. Now with AI assistants, I think it could be a great pick.