Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs pose an interesting problem for DSL designers

kirancodes.me

Post reply on HN
Print view View original

151–157 of 157 posts

Re: LLMs pose an interesting problem for DSL designers

#151

Post by chr15m »

csomar wrote: post #113
chr15m wrote: post #88

It's it just me or does the graph of LLM language performance versus training set size show the opposite of what they are saying? To me it looks flat, implying training set size has little influence on LLM performance in the language. For instance some niche languages appear to out-perform better known languages (with more variance in the niche language performance).

It is not only you, but I think it is only you and me. I've also skimmed through the comments and wondering if they are AI generated; or the people even read the article. The author essentially took a graph, and then claimed a different interpretation to reality. What the graph shows is that LLMs struggle with "hard" languages (Rust, Go, C#) with the exception of Ruby.

Good to know, thanks for your reply!

Re: LLMs pose an interesting problem for DSL designers

#152

Post by swyx »

romaniv wrote: post #125

The title should be "DSLs pose an interesting problem for LLM users". It is significant that LLMs in coding are being promoted based on a set of promises (and assumptions) that are getting instantly and completely reversed the moment the technology gets an iota of social adoption in some space. "Everyone can code now!" -> "Everyone must learn a highly specialized set of techniques to prompt, test generated code, etc.…

we had this with types too. fanaticism isn't unique to llms

Re: LLMs pose an interesting problem for DSL designers

#153

Post by ewoodrich »

jrmg wrote: post #123
NathanKP wrote: post #12

Good to see more people talking about this. I wrote about this about 6 months ago, when I first noticed how LLM usage is pushing a lot of people back towards older programming languages, older frameworks, and more basic designs: https://nathanpeck.com/how-llms-of-today-are-secretly-shapin... To be honest I don't think this is necessarily a bad thing, but it does mean that there is a stifling effect on fresh new DSL's…

It’s not just new frameworks, it’s new features. Good luck getting a LLM to write code that uses iOS 26 features, for example. I’m not convinced simply getting the LLM to inject documentation about the features will work well (perhaps someone has studied this?) because the reason they’re good at doing ‘well known’ things is the plethora of actual examples they’re trained on.

I have been dealing with an extremely frustrating issue after moving from a very popular but now deprecated package to its supported successor. The API looks deceptively similar in terms of method names which is causing endless grief for LLM pattern matching. Even when I provide the exact documentation needed for the new API the massive amount of training material scattered across the internet for the old package version dominates and has zero effect even after one reply into the conversation.

I've tried Gemini 2.5 Pro, o3 and Claude 4 and they have their own unique tone of confidently wrong, deception about reasoning and gaslighting but produce the same result. I wasted way too much time trying to figure out a zero shot prompting strategy before I just rewrote the whole thing by hand.

Re: LLMs pose an interesting problem for DSL designers

#154

Post by usrbinbash »

nothrabannosir wrote: post #119
usrbinbash wrote: post #114

Good. I'll chalk that up as one of the positive effects LLMs have on the software development environment (god knows there are few enough). DSL proliferation is a problem. I know this is not something many people care to hear, and I symphasize with that. Smart people are drawn to complexity and elegance, smart people like building solutions, and DSLs are complex and elegant solutions. I get it. Problem is: Too many s…

This feels like survivorship bias. Many of those older tools seem like they were once fancy new DSLs. We just respect them now as established, because they've been around for so long. But for every one thousand awkward DSLs that didn't make it, one new tool emerged which lifts software development to a new level. Would you say the same about a parallel universe where LLMs were introduced in 1960?

> Many of those older tools seem like they were once fancy new DSLs.

I think no one ever called Python or C or Rust, or Java or Go a "domain specific language".

> We just respect them now as established, because they've been around for so long.

No, we "respect" them because they are general purpose languages that you can do anything with.

> But for every one thousand awkward DSLs that didn't make it, one new tool emerged which lifts software development to a new level.

Please, do list some DSLs that managed to "lift software development to a new level". And again: A General Purpose Language is not a DSL.

Re: LLMs pose an interesting problem for DSL designers

#155

Post by daxfohl »

seanmcdirmid wrote: post #150
daxfohl wrote: post #141

Earlier quoted context omitted.

Though now that I say that, I wonder if maybe the approach should be, instead of specifying the onboarding procedure in terms of steps, maybe it would be better to specify a goal, the constraints, and some optimization function, and allow the AI to figure out what the workflow steps should be. So it would act higher level, more as a constraint solver than a code generator. That could be easier to interact with, allow…

I think the magic key might be LLMs that can build and test code, and then revise it and test it again, revise tests, etc...all while accepting feedback from human users about what the requirements really are (because they aren't going to be complete at the beginning). We might already be there with LLMs and we just haven't figured out the workflow yet (it isn't just one LLM activation, for sure, it has to be a serie…

Yes. I tried the prolog style requirements last night too and it wasn't great either. Some things are easier to say in "rules" and some things are easier to say in "steps". Trying to get all the way to "rules" ends up sounding like Yoda talk in places. Where with "steps" you feel the need for a GOTO, with "all rules" you feel the need for the infamous COMEFROM keyword in places.

So, yeah, I think what you say is the only logical conclusion. Waterfall development doesn't work well for humans, and it's probably not going to work for AI either. In fact, spending a ton of time formalizing a huge requirements doc is going to be even more of an antipattern with AI. When you can just have a conversation, hammer out details, identify nuances, plan for the future, all while seeing the feature in real time, that's going to be a million times more productive. The requirements doc can come with a full demo and a green button to launch it into production immediately after (or during) the review meeting.

So yeah, we'll probably never have "English as code", but rather an iterative process that leads to both English and code descriptions of the feature.

As far as DSLs, I think they still fit. If you see a pattern that makes sense to encode in a DSL, the LLM will happily do that, and doing so could help the AI optimize its limited context and make the invariants of the domain much more explicit.

Fun times.

Re: LLMs pose an interesting problem for DSL designers

#156

Post by nothrabannosir »

usrbinbash wrote: post #154
nothrabannosir wrote: post #119

Earlier quoted context omitted.

This feels like survivorship bias. Many of those older tools seem like they were once fancy new DSLs. We just respect them now as established, because they've been around for so long. But for every one thousand awkward DSLs that didn't make it, one new tool emerged which lifts software development to a new level. Would you say the same about a parallel universe where LLMs were introduced in 1960?

> Many of those older tools seem like they were once fancy new DSLs. I think no one ever called Python or C or Rust, or Java or Go a "domain specific language". > We just respect them now as established, because they've been around for so long. No, we "respect" them because they are general purpose languages that you can do anything with. > But for every one thousand awkward DSLs that didn't make it, one new tool eme…

I wasn’t talking about gp languages at all. I’m not the one to whom you need to repeat that ;)

- Makefiles

- regular expressions

- m4

- awk

- sed commands

- jinja templates

- jq

- nix

I have a love-hate relationship with many of these but I sure am happy that they tried. Whatever sucks about them is improved by an even better language, not by regressing to just using general purpose languages everywhere.

PS I put awk in there because you’re not the boss of me. It’s a dsl for text manipulation which happens to be Turing complete. If you write any serious program in awk that isn’t some form of text manipulation, it will make the front page of HN, that’s how rare that is. And to TFA’s point: awk excels at the niche for which it was developed—in a world where LLMs came before awk, it would be very hard to get awk off the ground. That is a net loss. That is the point of TFA.

You’re not wrong about the cost. But you’re wrong about the benefit.

Re: LLMs pose an interesting problem for DSL designers

#157

Post by usrbinbash »

nothrabannosir wrote: post #156
usrbinbash wrote: post #154

Earlier quoted context omitted.

> Many of those older tools seem like they were once fancy new DSLs. I think no one ever called Python or C or Rust, or Java or Go a "domain specific language". > We just respect them now as established, because they've been around for so long. No, we "respect" them because they are general purpose languages that you can do anything with. > But for every one thousand awkward DSLs that didn't make it, one new tool eme…

I wasn’t talking about gp languages at all. I’m not the one to whom you need to repeat that ;) - Makefiles - regular expressions - m4 - awk - sed commands - jinja templates - jq - nix I have a love-hate relationship with many of these but I sure am happy that they tried. Whatever sucks about them is improved by an even better language, not by regressing to just using general purpose languages everywhere. PS I put awk…

I think we're talking past each other. You may notice that I haven't condemned DSL in general. I condemn coming up with a DSL when using GP-Language would suffice as the easier solution.

And not to put too fine a point on it, but I have seen many awk scripts, elaborate sed-constructs (I refuse to call them scripts) and jq expressions, that would have been a lot simpler, easier to understand, and easier to maintain, if someone had just sat down and rewrote them in Python, or even in Go. They have their uses, but when people start developing what can be called a small application in a DSL that happens to be turing complete, they should just use a real programming language.

The same goes for many of Makefiles btw., some of which resemble an exceprt from the Necronomicon more than they do build instructions.

And btw. "survivorship bias" doesn't fit what's happening here at all, because the only reason we still use sed, or awk, is that they proved useful for a long time, where many other DSL did not.

Post reply on HN