When you're working on something difficult that requires a model to reason intelligently, lower level and strongly typed languages often outperform on the same problems [0]. We have a few hypotheses about why, with a moderately high correlation between performance and token density of the output program -- i.e. more token dense languages are more difficult for programs to reason about. Most models come up with the le…
Use boring languages with LLMs
111–120 of 180 posts
Re: Use boring languages with LLMs
#112I disagree. "Boring" languages leave a lot of assumptions in code, which will start to compound the more changes model (and programmers) make to the code. The more assumptions I can move to compile time the better models are at dealing with emerging complexity. I would go the other way with LLMs and I wish for liquid types and effects in Rust to make type specifications even more strict. P.S. effects and liquid types…
Sounds like OxCaml is pretty close to what you want. You get access to similar capabilities as Rust, but also stricter typing and an (optional) effect system. I don't know of an equivalent to Liquid Types, but it seems like the same approach that worked for Haskell would work naturally in OxCaml.
Which is why I think it's silly to suggest creating a new language "for agents". Unless one or more of the frontier AI companies commit to creating a language and the training corpus for a new language, there's no good way to bootstrap a language that is ideal for agents. You need the huge pile of high quality code as a prerequisite for a language being good for agents. And, the argument applies similarly poorly for some language that looks like it has a good shape for agents, if it doesn't have a lot of human written code from the past decade or whatever. It's not a good language for agents unless agents already know the language really, really, well because of a huge pile of code in that language in its training data.
Re: Use boring languages with LLMs
#113When you're working on something difficult that requires a model to reason intelligently, lower level and strongly typed languages often outperform on the same problems [0]. We have a few hypotheses about why, with a moderately high correlation between performance and token density of the output program -- i.e. more token dense languages are more difficult for programs to reason about. Most models come up with the le…
That's a very interesting page, but the language ranking is wildly different for "average percentage" (python bottom) and "success rate" (python second). Sounds like there is some subtly about this.
For one-shot responses, the majority of failures are environmental/syntax, which naturally favors interpreted languages. For longer agentic coding sessions, models solve the environment issues quickly and it becomes a fair comparison of who comes up with the smarter solution. You can filter for that here: https://gertlabs.com/rankings?mode=agentic_coding
Re: Use boring languages with LLMs
#114Re: Use boring languages with LLMs
#115Earlier quoted context omitted.
I know Clojure misses the mark for you in some major areas, but having a real, proper REPL for an agent to interact with makes for an extremely strong feedback loop. IMO, a strong candidate for an "agent endgame" language has this.
I love REPL-driven development, and exploratory "programming" via small snippets is likely part of the endgame (the more the agent strays outside its comfort zone, the more it's needed). It also looks like it paradoxically saves context, since piling up many small snippets is still better than trying to fix a one-shot gone horribly wrong. And yeah, if Clojure had a better static safety story, it would actually rank h…
Here's one of many practical examples I can give you - my WM on Mac is Yabai, it is hooked up to Hammerspoon, which can be scripted with Lua, which means I can use Fennel, which means I can have Lispy REPL. And from here, it is a bit difficult to explain the difference. The challenge is that the magic is invisible to people who haven't felt it.
The key insight is "the image vs the file". In most languages, your program is a description that gets turned into a running thing. The REPL is bolted on - a convenience wrapper around that same lint/compile/run/restore-the-state cycle. Python, Ruby, Lua, C#, etc. REPLs work that way. You're still fundamentally working with files that produce processes.
In a Lisp, the running system is the environment. There's no gap between "the code" and "the live thing". When I connect to Hammerspoon's Lua runtime via Fennel, I'm not sending scripts to a subprocess - I'm reaching into a living system and reshaping it while it runs.
The missing vocabulary here is "liveness", not "fast feedback" - that's a pale shadow of it. Liveness means the environment has no opinion about what's "done" versus "in progress". Everything is always mid-flight and accessible.
So I can actually reach out in live REPL session to let's say Slack app window and extract the data about every single element in the app, get the content, compare and continuously reiterate, without having to restart anything, without even saving the code - just pure data extraction without compiling, dealing with state changes, etc. I can interactively move the window, resize it, hide, or maximize it - all that programmatically. Imagine DevTools on steroids, only it works for everything, not just web apps.
Learning Lisp and Clojure allowed me to truly experience the genuine joy of programming, because it makes it feel like you're playing a video game. And now, can you even imagine what happens when you open up access to all that awesomeness and grant it to an LLM? Most people have zero idea what it feels and looks like, when you can point an agent to a REPL running in a k8s cluster and it introspects things on the fly, while you let another agent poke through the UI and they work as a team to fix something or develop a new feature.
> if Clojure had a better static safety story, it would actually rank high
This framing reveals an assumption - the primary value of a type system is catching errors early, and that Clojure is just a dynamically typed language that would be improved by adding that. In a Lisp image, "early" and "late" barely exist as meaningful categories. The feedback loop isn't compile-time vs runtime - it's just... now. You evaluate a form and you know immediately. The error is right there, in context, with the live data that caused it.
Static types are, in a real sense, a compensation for the gap I just described - the gap between the description and the running thing. When you can't easily inspect or reshape the live system, you want the compiler to tell you as much as possible before you cross that gap.
Clojure has Spec (which can do things most other type systems would struggle to express), it has instrumentation, it has a rich data inspection story. You said "debugger", Clojure has Flowstorm, which is one of the best debugger experiences I have ever encountered in any language, and I have used more than a few.
Re: Use boring languages with LLMs
#116When you're working on something difficult that requires a model to reason intelligently, lower level and strongly typed languages often outperform on the same problems [0]. We have a few hypotheses about why, with a moderately high correlation between performance and token density of the output program -- i.e. more token dense languages are more difficult for programs to reason about. Most models come up with the le…
I ran a little test with Go, TypeScript, Clojure, F#, Haskell, and Rust. Token count was roughly in the same ballpark, but it used the fewest for TypeScript, then Go. The rest required a bit more. Clojure always won in terms of lines of code though, generally coming in at 1/2 the size of the Go or Typescript solutions.
This article doesn't seem to mention it either https://martinalderson.com/posts/which-programming-languages...
even though states Clojure to be the most token efficient. I personally, honestly don't care much. In my opinion (using LLMs with multiple different languages), specifics of PLs don't matter to the point of stating a "clear winner". It's not the language that matters but the "stories you tell" with the language. And greatest stories sometimes told in the languages people long forgotten.
Re: Use boring languages with LLMs
#117Re: Use boring languages with LLMs
#118Rust, Elixir, and Go are the way to go for LLMs in my testing and experience, for this and other reasons
Here’s some of the reasons it’s so good with Elixir: https://dashbit.co/blog/why-elixir-best-language-for-ai
- Durable, 'enterprise grade' software patterns are baked into the runtime and into common, stable libraries that everyone uses
- You can use Ash, which pretty much entirely solves architectural considerations for many types of backends
- The tooling for inspecting and enforcing style (tidewave, credo, dialyzer, Dan's "vibe" ecosystem tools) is far beyond what I see in other ecosystems
- Ecosystem coverage for pretty much everything you need, including numerical software
- Excellent performance escape hatches (NIFs)
And, as has been shown in various benchmarks, agents are quite good at it.
My one problem in practice has been that getting tests right is hard. LLMs need a lot of cajoling to not build flaky tests with all the concurrency, and I find myself spending hours rewriting parts of the test suite once or twice a week.
Re: Use boring languages with LLMs
#119Anyone use this stuff with Delphi? I've been looking for tips for getting the best out agents for Delphi