DSLs Enable Reliable Use of LLMs
martinfowler.com
DSLs Enable Reliable Use of LLMs
1–10 of 88 posts
Re: DSLs Enable Reliable Use of LLMs
#2what does it do , can u explain more?
Re: DSLs Enable Reliable Use of LLMs
#3I agree. What is missing from the discussion about DSLs are the importance of tooling such as linters, LSPs, etc, to give the LLMs further context.
For example, charts/plots are often stringly typed with regards to column names and a DSL specific to plotting could give immediate feedback to an LLM.
Re: DSLs Enable Reliable Use of LLMs
#4Pretty much how I use LLMs these days. Even Chatbots are able to work with a 200 line spec for the DSL.
You'd think it wouldn't because, well, no training data, but a short spec is usually enough.
Re: DSLs Enable Reliable Use of LLMs
#5The gap I've hit generating GPU kernels with agents
code that compiles and runs fine but is slower than the baseline. Validator says pass, result is useless. Speed targets have to be part of the check, not just correctness
Re: DSLs Enable Reliable Use of LLMs
#6> The advantage holds while the DSL stays small and constrained enough that a few in-context examples can convey its usage. There is also a real upfront cost in designing and maintaining the language and its semantic model. The payoff is therefore concentrated in well-factored, genuinely constrained DSLs backed by a validator.
dsl stays small is doing all the heavy lifting here
the premise is that because of these few existing dsls (like PlantUML mentioned) my "new dsl" will be equally effective. PlantUML has millions of examples in the training data, my new dsls are not (specially if its not json/yaml or just function chain based). as the number of things that can mix and match increase you are basically looking at a whole system prompt just describing the new language.
this brings us to the second part. step 2: after dsl is 'planned' (note they use the java compiler), the dsl need to have a real compiler/executor, not just a validator. because if then you are going to ask the llm to "compile the dsl to implementation" we are back to square 1.
Re: DSLs Enable Reliable Use of LLMs
#7> The advantage holds while the DSL stays small and constrained enough that a few in-context examples can convey its usage. There is also a real upfront cost in designing and maintaining the language and its semantic model. The payoff is therefore concentrated in well-factored, genuinely constrained DSLs backed by a validator. dsl stays small is doing all the heavy lifting here the premise is that because of these fe…
I’ve had good luck with LLMs and ad hoc DSLs, as well as much less common DSLs like liquidsoap’s stream management DSL.
I don’t think there’s magic in DSLs, I just think LLMs respond well to clear, simple structure.
Compilation / execution is often true, but not necessary. DSLs can be entirely declarative and used just for gating the stages of a multi-step workflow with checkpoints that have more structure than natural language.
Re: DSLs Enable Reliable Use of LLMs
#8The gap I've hit generating GPU kernels with agents code that compiles and runs fine but is slower than the baseline. Validator says pass, result is useless. Speed targets have to be part of the check, not just correctness
"Performance is the most important feature"
Re: DSLs Enable Reliable Use of LLMs
#9The gap I've hit generating GPU kernels with agents code that compiles and runs fine but is slower than the baseline. Validator says pass, result is useless. Speed targets have to be part of the check, not just correctness
> but is slower than the baseline. Validator says pass, result is useless
You should read this blog, they cover this exact scenario - https://www.weco.ai/blog/first-evidence-of-recursive-self-im...
> One domain that suffers from this particularly is GPU kernel engineering. We adopt our previous idea for detecting reward hacking from SpecBench and apply that to a set of KernelBench tasks, measuring whether the speedup the agent reports on the unit tests actually survives in the end-to-end workload (e.g. model training). A kernel counts as reward hacking if less than half of its claimed speedup survives there, including outright slowdowns and failures.
Re: DSLs Enable Reliable Use of LLMs
#10[flagged]