Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DSLs Enable Reliable Use of LLMs

martinfowler.com

Post reply on HN
Print view View original

11–20 of 88 posts

Re: DSLs Enable Reliable Use of LLMs

#11

Post by billyp-rva »

codegladiator wrote: post #6

> The advantage holds while the DSL stays small and constrained enough that a few in-context examples can convey its usage. There is also a real upfront cost in designing and maintaining the language and its semantic model. The payoff is therefore concentrated in well-factored, genuinely constrained DSLs backed by a validator. dsl stays small is doing all the heavy lifting here the premise is that because of these fe…

> the premise is that because of these few existing dsls (like PlantUML mentioned) my "new dsl" will be equally effective. PlantUML has millions of examples in the training data, my new dsls are not (*specially if its not json/yaml* or just function chain based)

I can confirm that having a DSL that is json/yaml helps a ton. Kind of like static type checking, it eliminates entire swaths of syntactical errors, allowing the LLM to focus on the semantics.

> because if then you are going to ask the llm to "compile the dsl to implementation" we are back to square 1.

I think this is an edge case; 99% of the time you (and/or the LLM) would have access to the implementor so it wouldn't need to do this.

Re: DSLs Enable Reliable Use of LLMs

#13

Post by andy_ppp »

What is the general consensus on Martin Fowler - I worked with Thought Works and they were obsessed with overcomplicating everything, but maybe that is just agency in general?

I think it goes without saying that the biggest fight we have as developers is keeping things as simple as possible when most external factors encourage complexity, especially LLMs.

Re: DSLs Enable Reliable Use of LLMs

#14

Post by codegladiator »

brookst wrote: post #7
codegladiator wrote: post #6

> The advantage holds while the DSL stays small and constrained enough that a few in-context examples can convey its usage. There is also a real upfront cost in designing and maintaining the language and its semantic model. The payoff is therefore concentrated in well-factored, genuinely constrained DSLs backed by a validator. dsl stays small is doing all the heavy lifting here the premise is that because of these fe…

I’ve had good luck with LLMs and ad hoc DSLs, as well as much less common DSLs like liquidsoap’s stream management DSL. I don’t think there’s magic in DSLs, I just think LLMs respond well to clear, simple structure. Compilation / execution is often true, but not necessary. DSLs can be entirely declarative and used just for gating the stages of a multi-step workflow with checkpoints that have more structure than natur…

> less common DSLs like liquidsoap’s stream management DSL

seems to be on github since 2008 so definitely in the training data. i am not talking about less or more common. either "your dsl" would need to look something like someone elses dsl (at this point is it your dsl?) or you need some way to get your dsls examples in the training data for the llm, or feed it in the prompt.

> LLMs respond well to clear, simple structure

and what a "clear simple structure" for a dsl is also quite not mentioned. clear and simple would be quite subjective based on the domain, the article says let the llm go in a loop trying to figure out the dsl for you.

> checkpoints that have more structure than natural language

if llm is at any point in the structured generation part then either you have a deterministic validator/compiler or you are back to reading/reviewing it manually, what can you trust ?

Re: DSLs Enable Reliable Use of LLMs

#15

Post by jdw64 »

andy_ppp wrote: post #13

What is the general consensus on Martin Fowler - I worked with Thought Works and they were obsessed with overcomplicating everything, but maybe that is just agency in general? I think it goes without saying that the biggest fight we have as developers is keeping things as simple as possible when most external factors encourage complexity, especially LLMs.

I'm curious too. In the Korean IT scene, his name is legendary. Because many articles reference his writings, and famous Korean IT YouTubers worship him. (I worship him too.)

So I do have some questions. First, I still enjoy reading ThoughtWorks Radar. For someone like me, who's in a vulnerable position far from the cutting edge of technology, it always helps me keep some level of synchronization with the tech world. But I'm curious whether this is just a perspective from Korea, or if it's the same in the West.

And as for the tendency to overcomplicate things—I think that even when something is implemented simply, the explanation often ends up being quite complex. Honestly, I find Fowler's writing easy to read

Re: DSLs Enable Reliable Use of LLMs

#16

Post by codegladiator »

billyp-rva wrote: post #11
codegladiator wrote: post #6

> The advantage holds while the DSL stays small and constrained enough that a few in-context examples can convey its usage. There is also a real upfront cost in designing and maintaining the language and its semantic model. The payoff is therefore concentrated in well-factored, genuinely constrained DSLs backed by a validator. dsl stays small is doing all the heavy lifting here the premise is that because of these fe…

> the premise is that because of these few existing dsls (like PlantUML mentioned) my "new dsl" will be equally effective. PlantUML has millions of examples in the training data, my new dsls are not (*specially if its not json/yaml* or just function chain based) I can confirm that having a DSL that is json/yaml helps a ton . Kind of like static type checking, it eliminates entire swaths of syntactical errors, allowin…

> DSL that is json/yaml helps a ton

it definitely does, and i would say json/yaml is not a dsl. this example of json/yaml keeps coming in the form of "DSL". i would say your configuration is not a dsl, it a declaration. llms are better at declarative stuff ? maybe but there are hardly that many of complex declarative frameworks.

PlantUML is a real dsl. not just declarative yaml.

Re: DSLs Enable Reliable Use of LLMs

#17

Post by NamlchakKhandro »

williamcotton wrote: post #3

I agree. What is missing from the discussion about DSLs are the importance of tooling such as linters, LSPs, etc, to give the LLMs further context. For example, charts/plots are often stringly typed with regards to column names and a DSL specific to plotting could give immediate feedback to an LLM.

It absolutely blows me away that there is still a significant amount of people out there kidding themselves that they're effectively using a coding harness .... and don't get this simple fact.

Re: DSLs Enable Reliable Use of LLMs

#18

Post by voidhorse »

I'm really starting to tire of people making broad, general claims about how LLMs work or how to use them with N = 1 or 2.

An LLM is a statistics machine for goodness sake. Basically any general claim about them needs to exploit the law of large numbers to be even remotely sensible. You cannot extrapolate from one-off behavioral successes. LLMs are not understanding anything in the way humans do. If they did, yeah, maybe you could extrapolate hard from small samples, but they don't work or understand things like we do. You need to show that the behavior you are documenting is an average behavior the LLM converges toward in the long run.

Re: DSLs Enable Reliable Use of LLMs

#19

Post by bdcravens »

I frequently blur the line between ad-hoc DSL and pseudocode, and just hand it off to the LLM. I want to get the thoughts out of my head as fast as possible, using whatever structure makes sense to me. Even if you know all of the code to be written, I think this is a huge win with LLMs, where your intent is more important than syntax.
Post reply on HN