DSLs are a waste of time

I agree. I wrote a long rant against DSLs here [0]. I won't repeat it here. This isn't an infrastructure DSL though, just a private DSL for our product.

Since then the situation hasn't gotten any better, but sort of stayed the same. Two people on our team have been solving bugs in this DSL for the last few months, rather minor ones. Tomorrow in fact I have a meeting with one of them about adding a sort of limited pass-by-reference semantics into the language (which currently doesn't allow pass-by-reference). I'm very torn about it, as I feel this is a waste of time, and we should ditch this DSL. Users complain rightly that it isn't a "real" programming language. Our "standard library" is very very lacking, for example there's no I/O.

We just added FFI 2 months ago. My team lead who was supposed to do this years ago didn't, and in fact effectively stopped developing it 2 years ago, feeling overwhelmed, and ended up leaving a couple of months ago.

What this language or rather ecosystem DOES do well is having an internal build system that is completely transparent to the users (unless there are bugs). Well, I designed this build system so I'm partial to it.

I had actually written the C# API which I wrote about at the end of [0], that was supposed to be a replacement of this DSL with a "real" language. However I got burned out a couple of months ago after my team lead left, and halted development. There was also a long series of heated discussions with some of my users, who wanted me to remove some of the (rather simple) guardrails I made, and give them the ability to build complex hierarchies above this API, an API which I designed for simplicity. I warned them those hierarchies would end up creating DLL hell and would make it necessary to build devops tooling and maybe even infrastructure as code [1], which IMO is silly for most of the internal projects, which contain at most 5-6 people. From 2 different organizational groups, but still. My users aren't developers per se, and have a more or less support role.

I'm still torn as to what the correct direction here. Long term our small team (only 2 full time developers and 3 part time) cannot support 2 half-assed non-standard DSLs, plus that C# API for our many internal users. Something gotta give.

[0] https://news.ycombinator.com/item?id=31205265

[1] There, I tied this response to the OP!

We have seen this trend in feature engineering. Some companies selling feature stores try to promote their DSL as the way to create features - instead of general purpose frameworks, such as Pandas, Spark, Flink, or event DBT/SQL.

In this case, DSLs may have a short-term wow factor, that you don't have to build an maintain feature pipelines, but you're always playing catchup when the latest feature engineering technique is not available in your DSL. And then, you have your developers careers to think of - do they want to put a DSL for feature engineering on their resume?

True! I think used Perl about that long, and probably Ruby after that. And those are (obviously) general purpose.

Stated in another way: standalone are sometimes bad, but embedded DSLs are almost always bad.

> the biggest players (MySQL, Postgres, Sqlite) have actually done a reasonably good job of making the 98% paths very good and 90% of the rest "good enough".

Exactly - Unless you are doing something really unusual (or wrong), the vendors of these engines have almost certainly encountered and optimized for some shape approximating your scenario.

If you want to get into some extreme ends of the practice, "ancient" engines like DB2 are some of the most capable. Many have been around longer than most developers today have been alive (myself included). That is a lot of optimization legacy to argue against. The halloween problem & iceberg meme comes quickly to mind. Why wouldn't you want to stick with something that has already dealt with all of that bullshit?