So, formal verification then...
DSLs Enable Reliable Use of LLMs
51–60 of 88 posts
Re: DSLs Enable Reliable Use of LLMs
#52The post would have been stronger and more useful focusing and expanding on the abstraction portion of his message as well as system and program design. Those elements were there but seem sidelined by the focus on DSL. For most developers on most projects, the key take away is we all have to both get good at system and program design especially around abstractions, shapes, responsilibility. Then we have to get good at inserting our selves in the AI loop to steer those decisions a lot in the beginning, regularly as the project kicks off and still more than we think as the project matures.
Re: DSLs Enable Reliable Use of LLMs
#53Logically this makes sense, but in practice it doesn’t, at least in my experience with SwiftUI. The LLM isn’t any better about generating/understanding it. While the DSL is more formal than natural language, it’s not what we’re communicating to the LLM with, so it’s advantages are washed away. And typical code is more strict/rigorous than DSLs so I think that’s why I see worse results, because a typical languages com…
I’ve had great luck building SwiftUI apps with GPT-5.5 (now GPT-5.6 Sol), Opus 4.8 and Fable 5. I’m just offering another data point, not suggesting on the effectiveness of DSLs for this. One line of thinking can be that frontier models are already powerful enough to brute force through it, but this might be a stronger indicator for success for smaller models.
No argument that it works, it definitely does. I just think the LLMs have a hard time with tweaks/modifications of any kind, more so than what I see even in a custom GUI toolkit I’ve built that it has zero training data on.
Asking for simple changes to SwiftUI interfaces I’ve one-shot is always a frustrating experience for me
Re: DSLs Enable Reliable Use of LLMs
#54Earlier quoted context omitted.
> less common DSLs like liquidsoap’s stream management DSL seems to be on github since 2008 so definitely in the training data. i am not talking about less or more common. either "your dsl" would need to look something like someone elses dsl (at this point is it your dsl?) or you need some way to get your dsls examples in the training data for the llm, or feed it in the prompt. > LLMs respond well to clear, simple st…
> or you need some way to get your dsls examples in the training data for the llm, or feed it in the prompt. I'm really not sure this is true. Recently for work I've been making changes in a system written in tcl in the early 2000s, including a custom dsl that has never escaped this company. The LLMs can write it fine. It wrote it almost acceptably on first sight, an hour or two of targeted test cases to extract a on…
> The LLMs can write it fine. It wrote it almost acceptably on first sight
you are probably talking of some coding harness which looked up the existing code base and then made it and not something like first prompt to llm "write xyz testcase in my systems company dsl in tcl from early 2000".
> LLMs can write it fine
sure, coding by examples is fine (it goes back to "system prompt describing the language"). but the claim we are arguing is reliability. did the llm generate 100% test case or code after reading your existing codebase, likely not as you mention it "almost acceptably on first sight".
i would probably be in denial if was suggesting llms not good at finding and fitting patterns.
Re: DSLs Enable Reliable Use of LLMs
#55Earlier quoted context omitted.
I’ve had good luck with LLMs and ad hoc DSLs, as well as much less common DSLs like liquidsoap’s stream management DSL. I don’t think there’s magic in DSLs, I just think LLMs respond well to clear, simple structure. Compilation / execution is often true, but not necessary. DSLs can be entirely declarative and used just for gating the stages of a multi-step workflow with checkpoints that have more structure than natur…
> less common DSLs like liquidsoap’s stream management DSL seems to be on github since 2008 so definitely in the training data. i am not talking about less or more common. either "your dsl" would need to look something like someone elses dsl (at this point is it your dsl?) or you need some way to get your dsls examples in the training data for the llm, or feed it in the prompt. > LLMs respond well to clear, simple st…
For open source llms you can give your grammar to the llm to help it decide what valid next tokens it can generate.
Re: DSLs Enable Reliable Use of LLMs
#56Earlier quoted context omitted.
> the premise is that because of these few existing dsls (like PlantUML mentioned) my "new dsl" will be equally effective. PlantUML has millions of examples in the training data, my new dsls are not (*specially if its not json/yaml* or just function chain based) I can confirm that having a DSL that is json/yaml helps a ton . Kind of like static type checking, it eliminates entire swaths of syntactical errors, allowin…
> DSL that is json/yaml helps a ton it definitely does, and i would say json/yaml is not a dsl. this example of json/yaml keeps coming in the form of "DSL". i would say your configuration is not a dsl, it a declaration. llms are better at declarative stuff ? maybe but there are hardly that many of complex declarative frameworks. PlantUML is a real dsl. not just declarative yaml.
I would say any JSON or YAML that carries semantics with the syntax is a language. It may be a configuration language rather than a programming language, but it’s a language rather than just generic data for some program to slurp.
Re: DSLs Enable Reliable Use of LLMs
#57> The advantage holds while the DSL stays small and constrained enough that a few in-context examples can convey its usage. There is also a real upfront cost in designing and maintaining the language and its semantic model. The payoff is therefore concentrated in well-factored, genuinely constrained DSLs backed by a validator. dsl stays small is doing all the heavy lifting here the premise is that because of these fe…
> the premise is that because of these few existing dsls (like PlantUML mentioned) my "new dsl" will be equally effective. PlantUML has millions of examples in the training data, my new dsls are not (*specially if its not json/yaml* or just function chain based) I can confirm that having a DSL that is json/yaml helps a ton . Kind of like static type checking, it eliminates entire swaths of syntactical errors, allowin…
Have you looked into YAMLScript?
Re: DSLs Enable Reliable Use of LLMs
#58Earlier quoted context omitted.
> it eliminates entire swaths of syntactical errors, allowing the LLM to focus on the semantics As I understood it, the idea the article presents is basically the opposite: The LLM is completely clueless about the semantics, but by constraining the language so that semantic errors are syntax errors, we can catch the LLM's bullshit with a simple validation step and make it try again until it produces valid syntax, whi…
I thought they got like a logit mask that removes illegal next tokens, at least in llama.cpp and GBNF.
You're probably right about that. It looks like llama.cpp lets you specify a grammar file or a JSON schema - is that what you mean? If the syntax of your DSL can be expressed that way, I suppose you wouldn't have to validate it externally and retry.
Re: DSLs Enable Reliable Use of LLMs
#59Earlier quoted context omitted.
its a specific type of language, not all language are the same , the nature of that specificity is the main point of the the claim made in this article
all languages are embedded in some domain, therefore having a domain does not make a language "a specific type of language"
Many languages are in fact designed and publicized as general-purpose programming languages.
Re: DSLs Enable Reliable Use of LLMs
#60Earlier quoted context omitted.
> or you need some way to get your dsls examples in the training data for the llm, or feed it in the prompt. I'm really not sure this is true. Recently for work I've been making changes in a system written in tcl in the early 2000s, including a custom dsl that has never escaped this company. The LLMs can write it fine. It wrote it almost acceptably on first sight, an hour or two of targeted test cases to extract a on…
> The LLMs can write it fine. It wrote it almost acceptably on first sight you are probably talking of some coding harness which looked up the existing code base and then made it and not something like first prompt to llm "write xyz testcase in my systems company dsl in tcl from early 2000". > LLMs can write it fine sure, coding by examples is fine (it goes back to "system prompt describing the language"). but the cl…
Well yes of course I was using a harness lol. We're talking about the application of tools to engineering problems. Getting an LLM to cold start on a completely novel to it language without prompting is a different problem that I don't have.