I guess if you love writing DSLs this is an unfortunate development, but for me it's more of a glass half full: I can have the AI spit out boilerplate I need to solve a problem instead of spending a week building a one-off DSL compiler.
LLMs pose an interesting problem for DSL designers
11–20 of 157 posts
Re: LLMs pose an interesting problem for DSL designers
#12Good to see more people talking about this. I wrote about this about 6 months ago, when I first noticed how LLM usage is pushing a lot of people back towards older programming languages, older frameworks, and more basic designs: https://nathanpeck.com/how-llms-of-today-are-secretly-shapin...
To be honest I don't think this is necessarily a bad thing, but it does mean that there is a stifling effect on fresh new DSL's and frameworks. It isn't an unsolvable problem, particularly now that all the most popular coding agents have MCP support that allows you to bring in custom documentation context. However, there will always be a strong force in LLM's pushing users towards the runtimes and frameworks that have the most training data in the LLM.
Re: LLMs pose an interesting problem for DSL designers
#13Python is just a beautiful, well-designed language - in an era where LLM's generate code, it is kind of reassuring that they mostly generate beautiful code and Python has risen to the top. If you look at the graph, Julia and Lua also do incredibly well, despite being a minuscule fraction of the training data.
But Python/Julia/Lua are by no means the most natural languages - what is natural is what people write before the LLM, the stuff that the LLM translates into Python. And it is hard to get a good look at these "raw prompts" as the LLM companies are keeping these datasets closely guarded, but from HumanEval and MBPP+ and YouTube videos of people vibe coding and such, it is clear that it is mostly English prose, with occasional formulas and code snippets thrown in, and also it is not "ugly" text but generally pre-processed through an LLM. So from my perspective the next step is to switch from Python as the source language to prompts as the source language - integrating LLM's into the compilation pipeline is a logical step. But, currently, they are too expensive to use consistently, so this is blocked by hardware development economics.
Re: LLMs pose an interesting problem for DSL designers
#14personally i think DSLs could be helpful if they are really good at:
1. explaining the syntax clearly
2. providing a fast checker that provides good error messages
3. prevents errors
LLMs seem pretty good at figuring out these things when given a good feedback loop, and if the DSL truly makes complex programs easier to express, then LLMs could benefit from it too. Fewer lines of code can mean less context to write the program and understand it. But it has to be a good DSL and I wouldn't be surprised if many are just not worth it.
Re: LLMs pose an interesting problem for DSL designers
#15FWIW, the core assertion here isn't even LLM-specific. DSL design leans heavily on the idea of an expert author who understands the underlying data model well already. That is no less true in the meatspace world than it is for an AI. DSLs look great if they let you write the code you already know how to write faster. DSLs look like noise to everyone else , including Gemini and Claude. I used to be a big DSL booster i…
> DSLs look like noise
Exactly right. Now that we're in the era of LLMs and Coding Agents it's never been more clear that DSLs should be avoided; because LLMs cannot reason about them as well as popular languages, and that's just a fact. You don't need to dig any further, to think about pros and cons, imo.
The fewer languages there are in the world (as a general rule) the better off everyone is. We do need a low level language like C++ to exist and a high level one like TypeScript, but we don't _need_ multiple of each. The fact that there are already multiple of each is a challenge to be dealt with, and not a goal what we reached on purpose.
Re: LLMs pose an interesting problem for DSL designers
#16Me thinking, is it time for a new 'programming' language for LLM to use instead of tools API calls? Something high level with loose grammar, in between natural language and strict programming. Then the backend, may be another smaller model, translates it in API calls. With this approach backend can be improved and updated much faster and cheaper then LLM model.
Re: LLMs pose an interesting problem for DSL designers
#17I do not use LLM and I continue to work in the older way.
Also, domain-specific stuff can still be useful sometimes, and other stuff involved with designing a programming language.
Re: LLMs pose an interesting problem for DSL designers
#18Python is just a beautiful, well-designed language - in an era where LLM's generate code, it is kind of reassuring that they mostly generate beautiful code and Python has risen to the top. If you look at the graph, Julia and Lua also do incredibly well, despite being a minuscule fraction of the training data. But Python/Julia/Lua are by no means the most natural languages - what is natural is what people write before…
mhhm yes yes. There's a thread of discussion that I didn't quite chose to delve into in the post, but there is something interesting to be found in the observation that languages that are close to natural language (Python being famous for being almost executable pseudo-code for a while) being easier for LLMs to generate.
Maybe designing new languages to be close to pseudo-code might lead to better results in terms of asking LLMs to generate them? but there's also a fear that maybe prose-like syntax might not be the most appropriate for some problem domains.
Re: LLMs pose an interesting problem for DSL designers
#19personally i think DSLs could be helpful if they are really good at: 1. explaining the syntax clearly 2. providing a fast checker that provides good error messages 3. prevents errors LLMs seem pretty good at figuring out these things when given a good feedback loop, and if the DSL truly makes complex programs easier to express, then LLMs could benefit from it too. Fewer lines of code can mean less context to write th…
Since DSLs are necessarily niche, you're never going to have much training data in that language to feed into LLM training. Sure this problem can be overcome, but you're just creating more work than the time saved by having humans code in DSLs.
Re: LLMs pose an interesting problem for DSL designers
#20One of my big concerns (a little tangential) is that LLMs will have the effect of fixing programming language design and the current language landscape into stone. This could occur in proportion to their use by programmers. The languages that LLMs do the best and have in their training data will be the languages programmers use, and getting any new language into LLM data sets will be very hard.