Dhall is advertised as a configuration language but you can do a tad more. I use it in my blog-engine to fit a use-case I found was poorly addressed by other approaches: small-cardinality datasets that benefit from type-checking and templating (e.g., list of notes, a photo gallery). I don't claim the idea is especially novel but I found the use case rare and interesting enough to write some explanations, design, and a demo here:
https://lucasdicioccio.github.io/dhall-section-demo.html
The Dhall Configuration Language
141–150 of 192 posts
Re: The Dhall Configuration Language
#142As much as I love Dhall, I think it's actually better to allow any turing complete programming language and require it to generate the config in a certain format (whatever it is, maybe even dhall). The only thing is needed to run the config-generation process in a sandbox and restrict how long it can run. If it takes too long then we stop it - the same we should do with a Dhall config as well, even if Dhall theoretic…
> The only thing is needed to run the config-generation process in a sandbox and restrict how long it can run. That's a very brittle restriction at best, and if implemented strictly on time, it's a very flaky on as well.
How so? Normally, generating a config, even a big one, is a matter of milliseconds even for slow languages.
We are not talking about applying the config - we are just talking about using some inputs (variables) and generating a string / text-file that we return to the entitiy that then validates and uses the config.
Not sure why that would be flaky.
Re: The Dhall Configuration Language
#143Earlier quoted context omitted.
Kubernetes ate their lunch, mostly.
No. Tools like Ansible are just wrappers around ssh when you need to run a command across many servers. They don't do anything k8s does.
I'd say partially right.
10 years ago people were running hundreds/thousands of VMs/Servers with full operating systems on them. They would often be long-lived and you needed puppet/ansible to update or run stuff on them.
With kubernetes deployment and updating your app/data is all native. The servers/nodes often run some sort of cut down operating system and are immutable.
Re: The Dhall Configuration Language
#144Earlier quoted context omitted.
CUE's author invented Borg configuration language or BCL since 2008. BCL code is the 3rd largest human written code in Google internal code base. In 2019 Aprial, there is 180M lines of BCL, while C++/Java sits at ~300M. BCL configuration's large scale use probably is beyond any other infra as code use cases known to human. And the learning and ideas over more than a decade, is manifested in CUE. Personally, this is e…
For the record, when I was an SRE at Google, most people I talked to hated BCL with a passion.
Same here. It did some cool stuff but really bent your mind.
Agree with the other comment that the amount of it at Goog should have a mixed interpretation. Obviously it was useful, but although Google systems were complicated, it still doesn't seem right that it's on the same magnitude as the main system langs.
Re: The Dhall Configuration Language
#145Earlier quoted context omitted.
> In 2019 Aprial, there is 180M lines of BCL, while C++/Java sits at ~300M. It's... not obvious to me that that's a good thing? The ratio of configuration code to code in the things being configured being that close makes me think that BCL is something that's ill-suited to what it's now being asked to do but there's too much of it to realistically replace. Maybe it's amazing and the problem it's solving is so complic…
The number of lines of BCL at Google says little-to-nothing about the efficacy of the language itself, it's more of a reflection of the complexity and scale of the _systems_ its used to configure.
It's it a good language? Unknown.
Can it be used to successfully build and maintain configuration at extremely large volume and complexity scales? Yes.
Also we're not talking about using this language, but its spiritual successor.
Re: The Dhall Configuration Language
#146I work on a deployment tooling team. In 2019/2020 we did a deep dive into Dhall vs. Jsonnet^1 for standardizing config and kubernetes templating across my company (Zendesk). We ended up going with Jsonnet (although some Dhall evangelists in the company have kept the dream alive!), which I think is a more approachable language for many, but Dhall has a lot of cool features and good things going for it. Jsonnet is far…
CUE's author invented Borg configuration language or BCL since 2008. BCL code is the 3rd largest human written code in Google internal code base. In 2019 Aprial, there is 180M lines of BCL, while C++/Java sits at ~300M. BCL configuration's large scale use probably is beyond any other infra as code use cases known to human. And the learning and ideas over more than a decade, is manifested in CUE. Personally, this is e…
CUE talks about unification, is that similar to prolog meaning of that term ?
Re: The Dhall Configuration Language
#147Relying on configuration files to glue together a huge stack of tools and services in order to produce an end product is a huge mistake. I say it's a mistake fully knowing that nearly everyone in the industry is doing it. Configuration is opaque. Undebuggable. Unmaintainable. By its very nature. No matter what "language" you use for it. You should strive to keep configurable things to an absolute minimum. Although it…
> Configuration is opaque. Undebuggable. Unmaintainable. By its very nature. No matter what "language" you use for it. What makes you think so? How do you define configuration? I mean, in the extreme case you can see Python as a configuration language for the behaviour of the Python interpreter. Does that make Python a configuration language? Where do you draw the line?
Configuration is some sort of information that is stored in a text file, for reading by one or more programs, where these programs cannot meaningfully operate without this critical information in the configuration files.
If you have ever worked in the web startup industry, their use is endemic in everything. You often cannot deploy any web applciation without properly understanding tons of configuration files.
Because everyone builds applications out of services that must be glued together, and these services often glue themselves together via configuration files. The python side of the application code needs to figure out how to connect to all the databases (postgres, redis, elastic, memcached ... etc). Each of these databases sometimes needs its own configuration before it can meaninfully operate. And because everyone is using Docker, you need docker files for all of your services. The list of things that need text based configuration fiels goes on and on.
Go to any web company and ask around to see who can meaningfully understand and edit or maintain these configuration files, and you will find the majority of the people have no idea what's going on inside them. It's very common that only two people have a decent understanding of all the configs.
Now to answer your core question:
Why are they opaque and undebuggable?
Because to understand the configuration files, you have to understand the programs they are inteded for and the environments they end up in, and the language they are written in.
For example, to understand a three-line docker compose configuration for ha_proxy, you have to understand the language of docker compose, and the way configuration files are aggregated. For example, you can define some VARIABLES in one config files and have them be available for reading by another docker compose config file in a totally different place. But you have to understand under what conditions which files are included together. For instance, you can have 10 files define different values for one VARIABLE and each of them runs in a different environment. You have to understand all of this. And we haven't even gotten to ha_proxy yet. Now you have to understand how ha_proxy is influenced by the configurations you have specified. Let's say it's telling it about another server to connect to, or a directory to serve static assets from. The path of this directory depends entirely on how other docker images are composed together, which in turn is usually influenced by the contents of many other docker compose config files that are - again - spread out across many files. You need to figure out _which_ config file is used to make a certain directory available at a certain location, etc etc.
There's no tool that can debug this mess. There's no substitute for deep and intimiate knowledge about all the details of the system.
It's very possible for the whole system to collapse when you make one tiny small mistake in one configuration file. There's no tool that can help you debug what's going on.
Re: The Dhall Configuration Language
#148Earlier quoted context omitted.
Yeah, I also really dislike this mixing up of indentation levels because of a language limitation. The "{" denotes a container, and the "," a separation between items of said container, for me it makes sense for them to be nested a level deeper
both symbols are delimiters, one can choose to align them together even when they are not the same character. Opening and closing braces aren't the same character either, but people have been aligning them for ages, I don't see a reason why commas, while being part of the same expression, should not follow the same principle.
For me it's not about being the same character (like you've mentioned, opening and closing braces aren't either), it's about commas and braces indicating different things in the hierarchy. Not to mention the symmetry breaking: an opening brace together with data in a line, then a lonely one at the bottom.
Re: The Dhall Configuration Language
#149Earlier quoted context omitted.
> In 2019 Aprial, there is 180M lines of BCL, while C++/Java sits at ~300M. It's... not obvious to me that that's a good thing? The ratio of configuration code to code in the things being configured being that close makes me think that BCL is something that's ill-suited to what it's now being asked to do but there's too much of it to realistically replace. Maybe it's amazing and the problem it's solving is so complic…
It's the configuration complexity clock again. https://mikehadlow.blogspot.com/2012/05/configuration-comple...
This is a really good mental image of what happens - thank you for sharing it.
Re: The Dhall Configuration Language
#150Earlier quoted context omitted.
IIUC, Dhall is pure and have no side effects
Unless you count the ability to do imports from around the web. But you can still make it side-effect-less by using caching and semantinc hashes on those imports?
these are side-effects performed by the compiler's runtime in a sandboxed environment. Dhall scripts are unable to perform side-effects on their own. This is a completely different setting compared to unrestricted side-effects available within a script written in general-purpose language.