Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DSLs Enable Reliable Use of LLMs

martinfowler.com

Post reply on HN
Print view View original

81–88 of 88 posts

Re: DSLs Enable Reliable Use of LLMs

#81

Post by brookst »

codegladiator wrote: post #29
brookst wrote: post #24

Earlier quoted context omitted.

Re: less common, I was just saying it doesn’t take millions off examples like PlanetUML. > what can you trust I wasn’t clear enough here — you’re responding to DSLs as an interface from non-deterministic LLMs to deterministic external systems. What I meant was using DSLs as intermediate checkpoints in multi-LLM processing. If you just flow natural language through 5 LLM calls, the last one may be getting something ve…

its served me okay. > Perfect and deterministic? No, of course not. Just an improvement and mitigation. exactly, not reliable, as the article tries to portray

Interesting … do you see reliable == infallible? I don’t. But certainly all software is imperfect, and LLMs doubly so.

Re: DSLs Enable Reliable Use of LLMs

#82

Post by codegladiator »

brookst wrote: post #81
codegladiator wrote: post #29

Earlier quoted context omitted.

its served me okay. > Perfect and deterministic? No, of course not. Just an improvement and mitigation. exactly, not reliable, as the article tries to portray

Interesting … do you see reliable == infallible? I don’t. But certainly all software is imperfect, and LLMs doubly so.

for me reliable = deterministically repeatable. if a llm has been able to do a task successfully and i ask it to do the same task again, i want the reliability that i get the same outcome (be if failure or success)

like if someone says "is this car reliable", i dont expect an infallible car. but if the cars third gear is 'you know sometimes it doesnt work', i wouldnt take that car out of city.

Re: DSLs Enable Reliable Use of LLMs

#83

Post by _superposition_ »

codegladiator wrote: post #14
brookst wrote: post #7

Earlier quoted context omitted.

I’ve had good luck with LLMs and ad hoc DSLs, as well as much less common DSLs like liquidsoap’s stream management DSL. I don’t think there’s magic in DSLs, I just think LLMs respond well to clear, simple structure. Compilation / execution is often true, but not necessary. DSLs can be entirely declarative and used just for gating the stages of a multi-step workflow with checkpoints that have more structure than natur…

> less common DSLs like liquidsoap’s stream management DSL seems to be on github since 2008 so definitely in the training data. i am not talking about less or more common. either "your dsl" would need to look something like someone elses dsl (at this point is it your dsl?) or you need some way to get your dsls examples in the training data for the llm, or feed it in the prompt. > LLMs respond well to clear, simple st…

The whole point is that a dsl syntax is constrained enough to fit in the context window and not rely on training data. LLMs are damn near built for this exact use case.

Re: DSLs Enable Reliable Use of LLMs

#84

Post by jazzypants »

billyp-rva wrote: post #21
codegladiator wrote: post #16

Earlier quoted context omitted.

> DSL that is json/yaml helps a ton it definitely does, and i would say json/yaml is not a dsl. this example of json/yaml keeps coming in the form of "DSL". i would say your configuration is not a dsl, it a declaration. llms are better at declarative stuff ? maybe but there are hardly that many of complex declarative frameworks. PlantUML is a real dsl. not just declarative yaml.

> and i would say json/yaml is not a dsl But you can have DSLs that are json/yaml, is my point. > PlantUML is a real dsl. PlantUML is a DSL that isn't json/yaml. That doesn't make it better, and you can make the argument that it is worse because the tooling around it won't be as good.

That's not a domain-specific language. You can use JSON and YAML in other domains. A DSL is defined by its syntax.

Re: DSLs Enable Reliable Use of LLMs

#85

Post by voidhorse »

Terretta wrote: post #72
voidhorse wrote: post #18

I'm really starting to tire of people making broad, general claims about how LLMs work or how to use them with N = 1 or 2. An LLM is a statistics machine for goodness sake. Basically any general claim about them needs to exploit the law of large numbers to be even remotely sensible. You cannot extrapolate from one-off behavioral successes. LLMs are not understanding anything in the way humans do. If they did, yeah, m…

you have a premise at the heart of that: > understanding anything in the way humans do i'm not sure it's clearly established LLMs can't be a model of some part of “the way humans do” ? to be more specific, i'd argue LLMs “understand” awfully similarly to a brilliant (polymath) with early dementia or Alzheimer's no executive function, no short term memory, and absent both of those, conversing with that person about th…

> to be more specific, i'd argue LLMs “understand” awfully similarly to a brilliant (polymath) with early dementia or Alzheimer'

At which point i'd argue that you're kind of refuting your own claim as most humans are not brilliant polymaths with early onset alzheimer's.

Even admitting these kinds of comparisons to edge case human mental experience, there are more differences than similarities, and the similarities are misleading and superficial. There are still a lot of differences, even when it comes to the physical structure of the brains neurons as compared to digital neural nets, and i think it's far more beneficial to try to understand these machine in their uniqueness and for what they are than to draw hasty and shallow comparisons. You could be forgiven for that, though, because computing is rife with people who love to draw hasty unjustified analogies for some reason (example: people were already likening the brain to a computer when we didn't even have working implementations of neural nets yet and a computer was literally just a small number of logic gates lol)

Re: DSLs Enable Reliable Use of LLMs

#86

Post by samatman »

codegladiator wrote: post #82
brookst wrote: post #81

Earlier quoted context omitted.

Interesting … do you see reliable == infallible? I don’t. But certainly all software is imperfect, and LLMs doubly so.

for me reliable = deterministically repeatable. if a llm has been able to do a task successfully and i ask it to do the same task again, i want the reliability that i get the same outcome (be if failure or success) like if someone says "is this car reliable", i dont expect an infallible car. but if the cars third gear is 'you know sometimes it doesnt work', i wouldnt take that car out of city.

This seems too strong to me. I agree with reliable as "repeatable", it's the determinism which seems excess to purpose.

If I have a great meal every time I go to a restaurant, that chef is reliably good. I do not require that the tomatoes on the salad be of identical number or placement, or cut along the precise same bias. Nor would I count the sesame seeds on the bun.

Are LLMs reliable by that standard? Sometimes yes, for some tasks. The envelope of tasks for which they're reliable is continuing to grow.

Re: DSLs Enable Reliable Use of LLMs

#87

Post by orbitalventures »

I Agree 100% with this approach, we are discovering that discrete languages are exceptionally good for LLM to deal with them, classic verbs and recommended simple sentences, we have been having quite a good time using it as replacement of Terraform for managing infrastructure and the results are quite promising.

Re: DSLs Enable Reliable Use of LLMs

#88

Post by tingletech »

ptx wrote: post #58
tingletech wrote: post #43

Earlier quoted context omitted.

I thought they got like a logit mask that removes illegal next tokens, at least in llama.cpp and GBNF.

You're probably right about that. It looks like llama.cpp lets you specify a grammar file or a JSON schema - is that what you mean? If the syntax of your DSL can be expressed that way, I suppose you wouldn't have to validate it externally and retry.

the grammar support/json-schema applies to a whole response, but I was under the impression a similar logit guide rail can be applied when a tool call is opened.
Post reply on HN