Universal configuration language

The problem is when you want to base the "value" part of any of these key/value pairs off some other value, you can't compute any new value - and you wind up duplicating variables in the configuration files (or worse, over several configuration files). This leads to someone inventing a new configuration-to-configuration converter to do what could be done in a macro.

Key/Value pairs work in a rather limited portion of software, but most configuration formats are calling out for the ability to compute. Because they rarely work in practice, everyone forks the format to add their pet features, until some committee comes along and suggests "I know, I'll add all of your pet features into a universal format" - this thinking brought us to XML. Yaml, JSON and name-your-shitty-markup are continuations of this absurd line of thinking.

When TS says Lisp, he doesn't necessarily mean "configure the world in common lisp", but he's talking about S-expressions - which are a 'universal' way of encoding trees as text (without the element/attribute ambiguity), which you can chose to either treat as data or as code. The in memory representation of parsed s-expressions is equivalent to their textual representation (homoiconicity), which means that you can write code to operate on these structures using only the knowledge of the text layout, and not some extra knowledge your programming language might use for encoding it (ie, objects).

A configuration format using S-expressions need not be turing complete, as you can specify what should be data and what wants evaluating as code, if anything. You can place limits on what you want to be able to compute, by validating the input before evaluating it. As others have stated, the focus of configuration formats needs shifting from "syntactic flavor of the year" to proper validation of input. And the quickest path to validation of input is one where the parsing is automated - because Lisp does it for you.

The problem is there is a tendency for configuration languages to become Turing complete as they add more and more features. It would be preferable if they just started with something Turing complete. See here:

>Most projects seem to start out small with a few config items like where to write logs, where to look for data, user names and passwords, etc. But then they start to grow: features start to be able to be turned on or off, the timings and order of operations start to be controlled, and, inevitably, someone wants to start adding logic to it (e.g. use 10 if the machine is X and 15 if the machine is Y). At a certain point the config file becomes a domain specific language, and a poorly written one at that.

https://stackoverflow.com/questions/648246/at-what-point-doe...

I like the idea of using Lua as a config language because it's pretty simple, lightweight, and can be sandboxed easily.

My first and biggest issue with computation in configuration files is the moment you need to / want to manage that configuration with a tool.

In 30+ years I've never come across a case where logic has been necessary in configuration beyond very simple branching / inclusion logic, if that.

I am, I'm sorry to say, unconvinced. I'm not keen on 'magic' behaviours like auto-converting of values to arrays -- while the behaviour of the document may be well specified globally, it's nice to be able to see what something's going to do based on immediate (or even no) context.

What I'm very much liking at the moment is the way Dropwizard uses Jackson's YAML (and by extension, JSON) parsing for configuration -- your configuration file maps 1-1 to a class in your application, and Jackson is configured to fail if you're either missing a field that's not marked as optional or you've got extra fields in your YAML that don't map to anything. Type-safety FTW!

On the subject of wanting to refer to earlier bits of configuration: if you need to use one value as part of another, your configuration system might not be exposing the right level of detail. Of course, you might not be able to change that.

This has some good ideas, but it's very similar to HOCON. I think it's more worthwhile to focus on improving the HOCON spec rather than building yet another configuration json superset.

https://github.com/typesafehub/config/blob/master/HOCON.md