Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DSLs Enable Reliable Use of LLMs

martinfowler.com

Post reply on HN
Print view View original

21–30 of 88 posts

Re: DSLs Enable Reliable Use of LLMs

#21

Post by billyp-rva »

codegladiator wrote: post #16
billyp-rva wrote: post #11

Earlier quoted context omitted.

> the premise is that because of these few existing dsls (like PlantUML mentioned) my "new dsl" will be equally effective. PlantUML has millions of examples in the training data, my new dsls are not (*specially if its not json/yaml* or just function chain based) I can confirm that having a DSL that is json/yaml helps a ton . Kind of like static type checking, it eliminates entire swaths of syntactical errors, allowin…

> DSL that is json/yaml helps a ton it definitely does, and i would say json/yaml is not a dsl. this example of json/yaml keeps coming in the form of "DSL". i would say your configuration is not a dsl, it a declaration. llms are better at declarative stuff ? maybe but there are hardly that many of complex declarative frameworks. PlantUML is a real dsl. not just declarative yaml.

> and i would say json/yaml is not a dsl

But you can have DSLs that are json/yaml, is my point.

> PlantUML is a real dsl.

PlantUML is a DSL that isn't json/yaml. That doesn't make it better, and you can make the argument that it is worse because the tooling around it won't be as good.

Re: DSLs Enable Reliable Use of LLMs

#22

Post by bob1029 »

The only thing which enables reliable use of LLMs are statistical techniques.

Even the most constrained and well-designed Disney world ride will break down in some embarrassing way every now and again. As you increase the # of parallel rides, the chances that at least one of them will touch the desired parts of the search space go up dramatically.

The fact that the major model providers keep publishing nano/mini/luna variants should be a massive hint that there's more to this than one big fat loop magically one-shotting everything.

Re: DSLs Enable Reliable Use of LLMs

#24

Post by brookst »

codegladiator wrote: post #14
brookst wrote: post #7

Earlier quoted context omitted.

I’ve had good luck with LLMs and ad hoc DSLs, as well as much less common DSLs like liquidsoap’s stream management DSL. I don’t think there’s magic in DSLs, I just think LLMs respond well to clear, simple structure. Compilation / execution is often true, but not necessary. DSLs can be entirely declarative and used just for gating the stages of a multi-step workflow with checkpoints that have more structure than natur…

> less common DSLs like liquidsoap’s stream management DSL seems to be on github since 2008 so definitely in the training data. i am not talking about less or more common. either "your dsl" would need to look something like someone elses dsl (at this point is it your dsl?) or you need some way to get your dsls examples in the training data for the llm, or feed it in the prompt. > LLMs respond well to clear, simple st…

Re: less common, I was just saying it doesn’t take millions off examples like PlanetUML.

> what can you trust

I wasn’t clear enough here — you’re responding to DSLs as an interface from non-deterministic LLMs to deterministic external systems.

What I meant was using DSLs as intermediate checkpoints in multi-LLM processing. If you just flow natural language through 5 LLM calls, the last one may be getting something very different from what it’s prompt is designed for.

But if you make the DSL a contract for handoff, results are much more stable.

Perfect and deterministic? No, of course not. Just an improvement and mitigation. But it’s served me well.

Re: DSLs Enable Reliable Use of LLMs

#25

Post by tclancy »

bob1029 wrote: post #22

The only thing which enables reliable use of LLMs are statistical techniques. Even the most constrained and well-designed Disney world ride will break down in some embarrassing way every now and again. As you increase the # of parallel rides, the chances that at least one of them will touch the desired parts of the search space go up dramatically. The fact that the major model providers keep publishing nano/mini/luna…

This is a statistical technique, effectively: it shrinks the problem space the LLM faces at each decision point. Ideally, it would be a way to turn a request from an open-world game into a game on rails. Given the DSL, my options are only X or Y at this point in the solution.

Admittedly, that’s just improving the likelihood of getting a successful result, but if you mean 100% when you say “reliable”, that’s a false equivalence. No coder gets it right reliably either.

Re: DSLs Enable Reliable Use of LLMs

#26

Post by giovannibonetti »

This reminds me of this Bjarne Stroustrup's Rule (creator of C++): - For new features, people insist on loud, explicit syntax. - For established features, people want terse notation

Hillel Wayne [1] argues that the same applies for the differences between what beginners and experts desire from a language: Beginners need explicit syntax, experts want terse syntax.

In my mind, DSLs are related to that – a short notation to avoid repetition. And LLMs are the experts.

I wonder if Lisp with its powerful DSL-creating macros will enjoy more popularity in the near future.

[1] https://buttondown.com/hillelwayne/archive/stroustrups-rule/

Re: DSLs Enable Reliable Use of LLMs

#27

Post by wood_spirit »

Rather than DSLs I’ve found careful force tools results force the same kind of discipline in a more straightforward way to implement. So it’s normal to ask the llm to answer only yes or no and they are pretty good at following that instruction but it doesn’t scale so well. Whereas if the shape of the force tool call gives them more richness without giving them freedom to go off piste it scales to more nuanced results whilst also being trivial to parse.

Re: DSLs Enable Reliable Use of LLMs

#29

Post by codegladiator »

brookst wrote: post #24
codegladiator wrote: post #14

Earlier quoted context omitted.

> less common DSLs like liquidsoap’s stream management DSL seems to be on github since 2008 so definitely in the training data. i am not talking about less or more common. either "your dsl" would need to look something like someone elses dsl (at this point is it your dsl?) or you need some way to get your dsls examples in the training data for the llm, or feed it in the prompt. > LLMs respond well to clear, simple st…

Re: less common, I was just saying it doesn’t take millions off examples like PlanetUML. > what can you trust I wasn’t clear enough here — you’re responding to DSLs as an interface from non-deterministic LLMs to deterministic external systems. What I meant was using DSLs as intermediate checkpoints in multi-LLM processing. If you just flow natural language through 5 LLM calls, the last one may be getting something ve…

its served me okay.

> Perfect and deterministic? No, of course not. Just an improvement and mitigation.

exactly, not reliable, as the article tries to portray

Post reply on HN