LLMs pose an interesting problem for DSL designers

As AI systems improve, and especially as they add more 'self-play' in training, they might become really good at working in any language you can throw at some.

(To expand on the self-play aspect: when training you might want to create extra training data by randomly creating new 'fake' programming languages and letting it solve problems in them. It's just another way to add more training data.)

In any case, if you use an embedded DSL, like is already commonly done in Haskell, the LLMs should still give you good performance. In some sense, an 'embedded DSL' is just fancy name for a library in a specific style.

At the time, I had given in to Claude 3.5's preference for python when spinning up my first substantive vibe-coded app. I'd never written a line of python before or since, but I just let the waves carry me. Claude and I vibed ourselves into a corner, and given my ignorance, I gave up on fixing things and declared the software done as-is. I'm now the proud owner of a tiny monstrosity that I completely depend on - my own local whisper dictation app with a system tray.

I've continued to think about stack ossification since. Still feels possible, given my recent frustration trying to use animejs v4 via an LLMs. There's a substantial api change between animejs v3 and v4, and no amount of direction or documentation placed in context could stop models from writing against the v3 api.

I see two ways out of the ossification attractor.

The obvious, passive, way out: frontier models cross a chasm with respect to 'putting aside' internalized knowledge (from the training data) in favor of in-context directions or some documentation-RAG solutions. I'm not terribly optimistic here - these models are hip-shooters by nature, and it feels to me that as they get smarter, this reflex feels stronger rather than weaker. Though: Sonnet 4 is generally a better instruction-follower than 3.7, so maybe.

The less obvious way out, which I hope someone is working on, is something like massive model-merging based on many cached micro fine-tunes against specific dependency versions, so that each workspace context can call out to modestly customized LLMs (LoRA style) where usage of incorrect versions of your dependencies has specifically been fine-tuned out.

This is what I've been focused on last few years with a bit of Direction 3 via

  python -> smt2 -> z3 -> verified rust

requirements -> design -> design by contract -> subset of python -> gc capable language (a fork of golang with ML features?) -> low level compiled language (rust, zig or C++)

As you go from left to right, there is an increasing level of detail the programmer has to worry about. The trick is to pick the right level of detail for a task.

Previous writing: https://adsharma.github.io/agentic-transpilers/

This is so true.

A couple months ago I was trying to use LLMs to come up with code to parse some semi-structured textual data based on a brief description from the user.

I didn't want to just ask the LLM to extract the information in a structured format as this would make it extremely slow when there's a lot data to parse.

My idea was, why not ask the LLM to come with a script that does the job. Kind of "compiling" what the user asks into a deterministic piece of code that will also be efficient. The LLM just has to figure out the structure and write some code to exploit it.

I also had the bright idea to define a DSL for parsing, instead of asking the LLM to write a python script. A simple DSL for a very specific task should be better than using something like Python in terms of generating correct scripts.

I defined the DSL, created the grammar and an interpreter and I started feeding the grammar definition to the LLM when I was prompting it to do the work I needed.

The result was underwhelming and also hilarious at some times. When I decided to build a loop and feed the model with the errors and ask to correct the script, I ended up sometimes having the model returning back python scripts, ignoring completely the instructions.

As the author said, everything is easier in Python, especially if you are a large language model!