Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DSLs Enable Reliable Use of LLMs

martinfowler.com

Post reply on HN
Print view View original

41–50 of 88 posts

Re: DSLs Enable Reliable Use of LLMs

#41

Post by UncleEntity »

codegladiator wrote: post #6

> The advantage holds while the DSL stays small and constrained enough that a few in-context examples can convey its usage. There is also a real upfront cost in designing and maintaining the language and its semantic model. The payoff is therefore concentrated in well-factored, genuinely constrained DSLs backed by a validator. dsl stays small is doing all the heavy lifting here the premise is that because of these fe…

I just have them write the tools to write the DSL's to do the thing then (most of) the sloppy code stays in the generator and if all the different things depend on each other they don't go stale and whatnot. And let them design the DSL themselves for whatever task so it matches their 'internal concept' of how the things work.

Worked out pretty well so far but not really practical unless your goal is to make the tools to make the DSLs to make jitting VMs -- https://github.com/dan-eicher/BBQ kind of snowballed from "let's parse some binary files" to a way over the top toolkit for playing around with this stuff but, it's fun...

Re: DSLs Enable Reliable Use of LLMs

#42

Post by ActionHank »

BatteryMountain wrote: post #38

What I do for my one framework (in dotnet, strong typing & compiler) is to create claude skills (bash scripts and dotnet console apps) that claude can call to do various things within the framework. I have a template engine (for classes/db scripts/view engine templates), build tools, deployment tools, backup tools, infra docs - all of it works amazingly well. I built them in a standardized way so it's easy to chain t…

dotnet on linux friend, what are you using for templating?

Re: DSLs Enable Reliable Use of LLMs

#43

Post by tingletech »

ptx wrote: post #36
billyp-rva wrote: post #11

Earlier quoted context omitted.

> the premise is that because of these few existing dsls (like PlantUML mentioned) my "new dsl" will be equally effective. PlantUML has millions of examples in the training data, my new dsls are not (*specially if its not json/yaml* or just function chain based) I can confirm that having a DSL that is json/yaml helps a ton . Kind of like static type checking, it eliminates entire swaths of syntactical errors, allowin…

> it eliminates entire swaths of syntactical errors, allowing the LLM to focus on the semantics As I understood it, the idea the article presents is basically the opposite: The LLM is completely clueless about the semantics, but by constraining the language so that semantic errors are syntax errors, we can catch the LLM's bullshit with a simple validation step and make it try again until it produces valid syntax, whi…

I thought they got like a logit mask that removes illegal next tokens, at least in llama.cpp and GBNF.

Re: DSLs Enable Reliable Use of LLMs

#46

Post by rudedogg »

Logically this makes sense, but in practice it doesn’t, at least in my experience with SwiftUI. The LLM isn’t any better about generating/understanding it.

While the DSL is more formal than natural language, it’s not what we’re communicating to the LLM with, so it’s advantages are washed away. And typical code is more strict/rigorous than DSLs so I think that’s why I see worse results, because a typical languages compiler “catches” more mistakes, versus a DSL that’s easy to write but has lots of implicitness.

I’ve had the same journey experimenting with levels if abstraction too. Going lower, and exposing the LLM to the “full-stack” works much better than trying to build up abstractions it can’t see into without extra steps.

I don’t want to be too much of a hater, but these types of panacea/architecture posts are usually written by people who don’t work in the field, lack pressure or constraints, and get paid to goof around in castles of the mind. I would simply skip over it and hold my comments/opinions to myself, but they tend to have an outsized influence on software engineering practices.

Re: DSLs Enable Reliable Use of LLMs

#47

Post by skydhash »

That’s basically LISP 101. Before solving a problem, you build out the spare parts and the tooling. And then the software building feels like assembling lego blocks. Building a DSL or a good set of symbols (functions/classes/constants/enum/…) is the cornerstone of DDD.

The actual implementation details only matters at the coding stage. At the design stage, it’s better to define the glossary and its semantic.

Re: DSLs Enable Reliable Use of LLMs

#48

Post by giraffe_lady »

codegladiator wrote: post #14
brookst wrote: post #7

Earlier quoted context omitted.

I’ve had good luck with LLMs and ad hoc DSLs, as well as much less common DSLs like liquidsoap’s stream management DSL. I don’t think there’s magic in DSLs, I just think LLMs respond well to clear, simple structure. Compilation / execution is often true, but not necessary. DSLs can be entirely declarative and used just for gating the stages of a multi-step workflow with checkpoints that have more structure than natur…

> less common DSLs like liquidsoap’s stream management DSL seems to be on github since 2008 so definitely in the training data. i am not talking about less or more common. either "your dsl" would need to look something like someone elses dsl (at this point is it your dsl?) or you need some way to get your dsls examples in the training data for the llm, or feed it in the prompt. > LLMs respond well to clear, simple st…

> or you need some way to get your dsls examples in the training data for the llm, or feed it in the prompt.

I'm really not sure this is true. Recently for work I've been making changes in a system written in tcl in the early 2000s, including a custom dsl that has never escaped this company. The LLMs can write it fine. It wrote it almost acceptably on first sight, an hour or two of targeted test cases to extract a one page functional spec of the subset we needed and that was it. They are quite good at finding and fitting to patterns, go figure.

Re: DSLs Enable Reliable Use of LLMs

#50

Post by sheepscreek »

rudedogg wrote: post #46

Logically this makes sense, but in practice it doesn’t, at least in my experience with SwiftUI. The LLM isn’t any better about generating/understanding it. While the DSL is more formal than natural language, it’s not what we’re communicating to the LLM with, so it’s advantages are washed away. And typical code is more strict/rigorous than DSLs so I think that’s why I see worse results, because a typical languages com…

I’ve had great luck building SwiftUI apps with GPT-5.5 (now GPT-5.6 Sol), Opus 4.8 and Fable 5. I’m just offering another data point, not suggesting on the effectiveness of DSLs for this.

One line of thinking can be that frontier models are already powerful enough to brute force through it, but this might be a stronger indicator for success for smaller models.

Post reply on HN