Live data from Hacker News

Which programming languages are most token-efficient?

martinalderson.com

71–80 of 96 posts

Re: Which programming languages are most token-efficient?

#71
There is one class of languages missing in the comparison: Programming golf languages: E.g. Japt [1], Pyth [2] or Jelly [3].

Update: I noticed that the author mentions that "APL's famous terseness isn't a plus for LLMs." Isn't that just a design limitation of the LLM tokenizers?

[1]: https://github.com/ETHproductions/japt

[2]: https://github.com/isaacg1/pyth

[3]: https://github.com/DennisMitchell/jellylanguage

Re: Which programming languages are most token-efficient?

#72
I kinda (but not really because I don't much care about tokens and don't really know anything about models) wonder about Common Lisp. There's probably far fewer examples of CL code in any training sets than Clojure or Python or whatever, but it could still be somewhat interesting.

Re: Which programming languages are most token-efficient?

#73
post #61

Concatenative languages like Factor and Forth are very token-efficient in theory. Theoretically optimal for raw lexical density. No parentheses, no commas, no argument delimiters, just whitespace-separated words, but stack shuffling can add overhead for complex data flow, unless you use "locals" in Factor, for example. C is surprisingly efficient as well. Minimal keywords, terse syntax, single-character operators. No…

I understand your logic but I found LLM's to be quite strong at C#. It makes little mistakes and the mistakes seem related to the complexity of what I'm doing, not the language itself.

See https://news.ycombinator.com/item?id=46586312.

I cannot speak much for C#, but you may be right. Claude's Opus is really good.

Re: Which programming languages are most token-efficient?

#74

This confirms my personal experience with switching to Go from C# - despite the e framework and language being MUCH simpler, the code usually ends up the same length. C# often has a 'nice' and 'performant' way of doing things (for example, strings are nice, but they allocate and are UTF16, but ReadOnlySpan is faster for UTF8, and can reuse buffers), the performant syntax often ends up being very verbose, with the nic…

[dead]

Re: Which programming languages are most token-efficient?

#75
post #35

Earlier quoted context omitted.

The approaches used by Claude Code and Cursor are inefficient. It's possible to calculate a covering set for a piece of code and provide that to an agent directly via a tool, and it turns out that this can reduce context usage in SWE-bench style tasks by >90% over RAG and grep/read. If you're interested in learning more, https://github.com/sibyllinesoft/scribe

Like most LLM-made readme's and the six bajillion AI/agentic/llm tools now on Github I can barely get a grasp on what I'm looking at here, or how to use it practically. > Smart code bundler that turns repositories into optimized code bundles meeting a token budget in milliseconds Ok. So it's a tool, do I use it on my repo once? Then what? Do I use it as I go, does it sit somewhere accessible to something like Claude…

The main box on the readme should make it pretty clear. One tool call to get a covering set of a piece of code, versus wasteful grep/read/lsp/etc.

I'm not sure if you're being intentionally obtuse or you just don't have much of an attention span, but I'm not making any money off this so if you want to use 10x more tokens to get stuff done, by all means brother.

Re: Which programming languages are most token-efficient?

#77
post #35

Earlier quoted context omitted.

The approaches used by Claude Code and Cursor are inefficient. It's possible to calculate a covering set for a piece of code and provide that to an agent directly via a tool, and it turns out that this can reduce context usage in SWE-bench style tasks by >90% over RAG and grep/read. If you're interested in learning more, https://github.com/sibyllinesoft/scribe

Like most LLM-made readme's and the six bajillion AI/agentic/llm tools now on Github I can barely get a grasp on what I'm looking at here, or how to use it practically. > Smart code bundler that turns repositories into optimized code bundles meeting a token budget in milliseconds Ok. So it's a tool, do I use it on my repo once? Then what? Do I use it as I go, does it sit somewhere accessible to something like Claude…

This scribe tool seems to offer somewhat similar functionality to a Language Server and/or Cursor's chunked vector index.

The idea would seem to be to give instructions to your agent (Claude Code, etc) to use this tool to discover the chunks of code (not entire source files) it needs to look at to modify a particular function. You could put these instructions on how/when to use scribe someplace like .claude/rules/scribe.md

I assume this is meant to work as an override to Claude Code's normal operation where it reads entire source files into context (not sure on details as to how CC decides which files are relevant if developer hasn't explicitly told it), so if you asked CC to do something that matches the instructions you'd put in scribe.md it would run scribe, send the output (code chunks and file locations) to Claude AI, which would then base it's edit requests on that.

It's not obvious if this --covering-set command is the only one scribe currently supports, or if it has other ones to output code chunks relevant for other use cases.

Re: Which programming languages are most token-efficient?

#78
post #63

I program mostly in Clojure and I expected it to be near the top, as it tends to be very concise and expressive (qualities I really admire). I am getting excellent results from Claude Code (Opus 4.5), and I think this might be one of the reasons. I'm using Claude with a large code base and the token-efficiency of Clojure might help with fitting more into the context window.

I also program a lot in Clojure/Script. Do you also consider thinking token and the number of iterations in the token efficiency?

Re: Which programming languages are most token-efficient?

#80
post #35

Earlier quoted context omitted.

Like most LLM-made readme's and the six bajillion AI/agentic/llm tools now on Github I can barely get a grasp on what I'm looking at here, or how to use it practically. > Smart code bundler that turns repositories into optimized code bundles meeting a token budget in milliseconds Ok. So it's a tool, do I use it on my repo once? Then what? Do I use it as I go, does it sit somewhere accessible to something like Claude…

This scribe tool seems to offer somewhat similar functionality to a Language Server and/or Cursor's chunked vector index. The idea would seem to be to give instructions to your agent (Claude Code, etc) to use this tool to discover the chunks of code (not entire source files) it needs to look at to modify a particular function. You could put these instructions on how/when to use scribe someplace like .claude/rules/scr…

Scribe grew out of fixing all the problems with code bundlers like Repomix. The covering set feature is the thing that clearly sets it apart, the performance difference is extreme; up to 98% token use reduction on SWE-bench tasks. I lead with it because it's the place where I'm far ahead of other tools, people won't adopt something because it's slightly better, scribe is a step change.
Post reply on HN