Live data from Hacker News

The Configuration Complexity Clock (2012)

mikehadlow.blogspot.com

11–20 of 24 posts

Re: The Configuration Complexity Clock (2012)

#11
REALLY worth reading the Graham Poulter comment in the original post about the difference between spatial and temporal variations.

The idea that you can hard-code everything assumes there is no variation in deployments and use of a piece of software.

If that were the case, you're talking about a SUPER simple piece of software. So sure, hard code everything.

But as soon as you're talking about a piece of software that will be used in multiple locations and with possibly different release / roll-out schedules, "there will be issues".

Re: The Configuration Complexity Clock (2012)

#12
post #3

Good analysis but too timid in its conclusions. At every level of complexity, hard-coding a solution is the least evil option. Your codebase is a living expression of your business rules, and it's already using the best format you have for expressing your business domain logic (if this isn't true, use a better language). When the business rules change, the codebase should change. You already need to be able to deploy…

There is a tension between delivery speed and operational safety. The faster you deploy new service code, the faster you can take down your service because of a bug that slipped past your automated tests. This is why one-box deployments and traffic shifting and feature flags exist (which are increasingly config-driven).

And if you have multiple instances of your service running (e.g. regional endpoints), and you care about availability, then you'll be deploying sequentially to each endpoint and watching for anomalies before moving to the next endpoint.

There is still immense value in dynamic config in these scenarios.

Re: The Configuration Complexity Clock (2012)

#13
post #10

Earlier quoted context omitted.

> If you want to "reconfigure" your app while it's running I don't think anyone was implying having a config file obviated a program restart. I'd add that restarting an unchanged executable with an altered config file is going to be considerably faster than recompiling it. > why not give it some input If that's not a config file, then what are you suggesting?

> I'd add that restarting an unchanged executable with an altered config file is going to be considerably faster than recompiling it. Why should it be? We presumably have a VCS tag corresponding to the current version such that it would be very easy to check out the corresponding version and change only one file. Then an incremental rebuild with just that one file change should not take a significant amount of time.…

> an incremental rebuild [...] should not take a significant amount of time.

> But actually a config change is just as dangerous as a code change, and the same level of testing is usually appropriate

Points well made, unarguably so

> In my experience many - perhaps most - production outages tend to be caused by "config changes".

I have no comparable experience but it sounds all too plausible and I shall bear that strongly in mind henceforth.

> "Config" tends to not be treated the same way [as code] [...]

From there to the end is brilliant. Thanks so much for such an excellent response. I'm going to print this out and keep it.

Re: The Configuration Complexity Clock (2012)

#14
I once worked at a place that reached DSL on the clock; no one understood the undocumented DSL (not even the programmers supporting it) but a few power users. I advocated re-writing rules in Python, using modern CI/CD techniques to allay fears of hard-coding. But it was too big of a philosophy change. The counter argument was "We don't want end users writing code!" but of course they were already writing code, just in a non-Google-able language...

Re: The Configuration Complexity Clock (2012)

#15
post #12
post #3

Good analysis but too timid in its conclusions. At every level of complexity, hard-coding a solution is the least evil option. Your codebase is a living expression of your business rules, and it's already using the best format you have for expressing your business domain logic (if this isn't true, use a better language). When the business rules change, the codebase should change. You already need to be able to deploy…

There is a tension between delivery speed and operational safety. The faster you deploy new service code, the faster you can take down your service because of a bug that slipped past your automated tests. This is why one-box deployments and traffic shifting and feature flags exist (which are increasingly config-driven). And if you have multiple instances of your service running (e.g. regional endpoints), and you care…

> There is a tension between delivery speed and operational safety. The faster you deploy new service code, the faster you can take down your service because of a bug that slipped past your automated tests.

100% agreed - but people miss that this applies just as much to deploying a "config change" as it does to a code change. Particularly if you're talking about feature flags, a config change and a code change are effectively equivalent - so you should apply the same standard to both. (What that standard should be is still a tradeoff)

> This is why one-box deployments and traffic shifting and feature flags exist (which are increasingly config-driven).

> And if you have multiple instances of your service running (e.g. regional endpoints), and you care about availability, then you'll be deploying sequentially to each endpoint and watching for anomalies before moving to the next endpoint.

I would agree that feature flags are config-like - and view them as the same antipattern. Traffic shifting and sequential deployment are great ideas, but they work great when deploying a code change. Make the change, gradually deploy the new version, roll back if needed. What does dynamic config gain you in that scenario? Only more possibilities to make mistakes, IME (e.g. the new version looked great on the "canary" node, but actually that instance had a different config from the other nodes).

Re: The Configuration Complexity Clock (2012)

#16

REALLY worth reading the Graham Poulter comment in the original post about the difference between spatial and temporal variations. The idea that you can hard-code everything assumes there is no variation in deployments and use of a piece of software. If that were the case, you're talking about a SUPER simple piece of software. So sure, hard code everything. But as soon as you're talking about a piece of software that…

That comment describes a real issue that the main post overlooks. But the comment also overlooks the obvious solution. It says:

>While "temporal" variations can easily be hardcoded if you have a short release cycle, "spatial" variations are not so easily hardcoded: you end up maintaining a source branch for each active variant.

But that's not the only way. There's a way that we are all very familiar with for customizing a component for being used in different situations: Passing in different arguments to the constructor.

If you have two places you want to deploy a piece of software, which have different environments and need to, e.g., access resources at different paths, then just have two main() functions, each invoked by one of two different executables. Those different main() functions can then hardcode all the specific details of whatever place you're deploying to, in your normal programming language, without you having to create configuration files or anything.

Re: The Configuration Complexity Clock (2012)

#18
The driver here is needing to make changes to the behavior of the app in specified ways, faster than the release cycle. My advice is to go no further than key-value configuration settings, and keep a documented set of Postman requests in the repo to serve as your UI for developers to invoke.

This way you get the immediacy of being able to change prod behavior outside of the release cycle, the safety of knowing only your devs can make those changes, and the ability to easily build a real UI later if the hidden features become features you want visible to non-technical users or your customers.

A rules engine is where the descent into madness begins. Every single thing the rules engine tweaks needs to be an actual feature with actual RESTful routes dedicated. Overloading a configuration regime, which is only supposed to handle keys and values, into the key instrumentation for the entire application, bolts inevitably poorly-documented semantics onto the application.

Different devs or departments will see the two competing regimes and pick whichever one they like the most to add on to. You'll end up with two kingdoms at war. You want peace reigning throughout your empire.

Configuration is part of your application infrastructure. Rules engines generate competing semantics. Semantics are how the brain understands systems. You want one overarching paradigm, one source of truth for how things get done in your application.

Re: The Configuration Complexity Clock (2012)

#19
post #15
post #12

Earlier quoted context omitted.

There is a tension between delivery speed and operational safety. The faster you deploy new service code, the faster you can take down your service because of a bug that slipped past your automated tests. This is why one-box deployments and traffic shifting and feature flags exist (which are increasingly config-driven). And if you have multiple instances of your service running (e.g. regional endpoints), and you care…

> There is a tension between delivery speed and operational safety. The faster you deploy new service code, the faster you can take down your service because of a bug that slipped past your automated tests. 100% agreed - but people miss that this applies just as much to deploying a "config change" as it does to a code change. Particularly if you're talking about feature flags, a config change and a code change are ef…

Not all software is "your service" that you can "take down" to install changes; there are cases in which there are multiple deployments with essentially independent and different configuration data that cannot be part of program code.

A popular example: the Git client, which stores layered configuration files for repositories, users and installations including, for instance, local file names (such as diff tools) and user identities.

Git also serves as an example of how the line between configuration files and data can be blurred: it's customary to put in Git repositories .gitignore and other similar additional configuration files because it's useful and the software only cares about their presence, not their management.

Re: The Configuration Complexity Clock (2012)

#20
post #3

Good analysis but too timid in its conclusions. At every level of complexity, hard-coding a solution is the least evil option. Your codebase is a living expression of your business rules, and it's already using the best format you have for expressing your business domain logic (if this isn't true, use a better language). When the business rules change, the codebase should change. You already need to be able to deploy…

The best thing you can do here is that if you use "8080" in a bunch of places in your code, replace all the ones that mean the same thing with a constant (eg, what port I'm running on), and the ones that mean something else (eg, what port the auth token engine is running on) with a different constant. Better still, use the URL library from your ecosystem to do this, and pester your 3rd party library authors to stop parting out URLs. If I had a dollar for every URL interpolation bug I've had to fix...

That way when you have a customer whose antivirus is running on 8080, you can finally pull it out into config files, and you 'just' have to change the people who consume the constant to pull from config. Because you've already telegraphed the intent to do this.

A few jobs back I saw another side benefit to this: you can entice people to participate in improving the code. When code is really wrong people will ignore it. They can't be bothered to get invested in it. But when it's almost right they are often motivated to fix it the rest of the way. Either they see the potential and are inspired (new blood, trying to make their mark), or the unfulfilled potential grates on their sensibilities (crotchety veterans).

Post reply on HN