What the author appears to miss is that many existing IAC tools permit exactly this. CDK, CDKTF and Pulumi all use general purpose programming languages, so reusing parameter objects in the way that is described is trivial - indeed it is so close to second nature that I would not even think to write it down. Indeed, it's not uncommon to share functions that make such parameter objects via libraries in the package eco…
Adding a "real" programming language makes certain things easier, such as abstraction, but IMO they are too powerful for the task at hand. Do we really want an infrastructure description to be able to execute arbitrary code?
Rethinking Infrastructure as Code from Scratch
21–30 of 124 posts
Re: Rethinking Infrastructure as Code from Scratch
#22Cloud template definitions also have a lot of settings, but from what I can see, they are all different, all the time, for lots of good reasons. If I'm deploying a lot of different kinds of EC2 instances, I've got a whole bunch of settings that are going to be different for each type. Abstracting is a much different problem as a result. And it isn't just this moment in time, it's the evolution of the system over time, too. In code, overabstracting happens sometimes. In cloud architecture it is an all-the-time thing. It is amazingly easy to over-abstract into "hey this is our all-in-one EC2 template" and then whoops, one day I want to change the instance size for only one of my types of nodes, and now I either need to un-abstract that or add yet another parameter to my all-in-one EC2 template.
The inner platform effect is very easy to stumble into in the infrastructure code as a result, where you have your "all-in-one" template for resource X that, in the end, just ends up offering every single setting the original resource did anyhow.
By contrast, I've pondered the "focus on the links rather than the nodes" idea a few times, and there may be something there. However the big problem I see is that I like rolling up to a resource and having one place where either all the configuration is, or where there is a clear path for me to get to that point. Sticking with an instance just to keep things relatable, if I try to define an instance in terms of its relationship to the network, to the disk system, to the queues that it uses and the lambda it talks to and the autoscaling group it is a part of, now its configuration is distributed everywhere.
One possible solution I've often pondered is modifying the underlying configuration management system to keep track of where things come from, e.g., if you have a string that represents the name of the system you're creating, but it is travelling through 5 distinct modules on its way to the final destination, it would be great if there was a way of looking at the final resource and saying "where exactly did that name come from?" and it would tell you the file name and line number, or the set of such things that went into it. Then at least you could query the state of a resource, and rather than just getting a pile of values, you'd be able to see where they are coming from, dig into all the things that went into all the decisions, that might free you to do link-based configuration rather than node-based configuration. But you'd probably need an interactive explorer; if for instance the various links can configure the size of the underlying disk and you take the max() of the various sizes (or the sum or whatever), you'd need to be able to look at everything that went into the max and all the sources of those values; it's more complicated than just tracking atomic values through the system.
I've often wished for this even in just my small little configs I manage compared to some of you, and it is possible that this would be enough of an advantage to stand out in the crowd right now.
(I think the "track where values came from and how they were used in computation" could be retrofitted onto existing systems. "Focus on links rather than nodes" will require something new; perhaps something that could leverage an existing system but would require a new language at a minimum.)
Re: Rethinking Infrastructure as Code from Scratch
#23I use Azure. When I need to provision anything I have a powershell script that interacts with Azure CLI. My script sets up a new resource group for every service we create, logging, key vault, webapp/functions, and if needed some kind of data storage or queuing. In my powershell script I can via a variable indicate which environment I want to spin up: dev, staging, prod. I have one yaml file which is for my build and…
Re: Rethinking Infrastructure as Code from Scratch
#24this is great, but I would argue the biggest issue with infrastructure as code is this: the structure and syntax for AWS is entirely different from Azure is entirely different from GCP. Instead of abstracting to CSS, I would argue modeling what Bytecode did in java for multi-operating system, we should do for infrastructure of code. That way, you could easily replicate in different environments, free yourself from ve…
If you sit down with the terraform specifications for an AWS instance, a GCP instance, and an Azure instance, and start trying to write that harmonization, you will rapidly discover why for yourself. Even just trying to specify a network setup and putting an instance on the public internet is impossible to harmonize, without making something so lowest-common-denominator it is almost useless, let alone anything complicated.
Re: Rethinking Infrastructure as Code from Scratch
#25My problem with the current IAS systems is the state storage. It should not be needed! Instead, the IAS tool should introspect the systems it's managing and build the necessary state on the fly.
This does not work. Say I have resource A with property X=1 I define in IAC. Someone comes along and modifies X=2 outside of state. With your way, the IAC tool would see that change and think it was naturally part of the desired state, whereas stored state will catch the drift. And before anyone says “well dont modify outside of IAC” I say 1) that’s often impractical and 2) sometimes automation can modify resources o…
Re: Rethinking Infrastructure as Code from Scratch
#26My problem with the current IAS systems is the state storage. It should not be needed! Instead, the IAS tool should introspect the systems it's managing and build the necessary state on the fly.
This does not work. Say I have resource A with property X=1 I define in IAC. Someone comes along and modifies X=2 outside of state. With your way, the IAC tool would see that change and think it was naturally part of the desired state, whereas stored state will catch the drift. And before anyone says “well dont modify outside of IAC” I say 1) that’s often impractical and 2) sometimes automation can modify resources o…
Re: Rethinking Infrastructure as Code from Scratch
#27I use Azure. When I need to provision anything I have a powershell script that interacts with Azure CLI. My script sets up a new resource group for every service we create, logging, key vault, webapp/functions, and if needed some kind of data storage or queuing. In my powershell script I can via a variable indicate which environment I want to spin up: dev, staging, prod. I have one yaml file which is for my build and…
Have you ever actually used terraform? You execute it from your own local computer, or from CI/CD. It runs in a compute resource you own, not the cloud provider.
Re: Rethinking Infrastructure as Code from Scratch
#28Did that guy just suggest that to make infrastructure-as-code easier to understand we should make it more like CSS ?
It seems like this will just amplify mistakes when a lowly dev tries to increase the available RAM of their resource and instead doubles the entire RAM allotment of a resource type for the entire enterprise.
Re: Rethinking Infrastructure as Code from Scratch
#29this is great, but I would argue the biggest issue with infrastructure as code is this: the structure and syntax for AWS is entirely different from Azure is entirely different from GCP. Instead of abstracting to CSS, I would argue modeling what Bytecode did in java for multi-operating system, we should do for infrastructure of code. That way, you could easily replicate in different environments, free yourself from ve…
This!
We're essentially writing locked-in vendorscript. What we want is an actual infrastructure language. One that lets us write once and deploy anywhere (nods to Java).
That would also allow us to standardize the way additional tooling (monitoring, logging, etc.) hooks into everything. It would allow us to easily deploy to new environment types as they become available (I keep hearing about the wonders of WASM). It would allow standardized ways of doing ops testing.
Re: Rethinking Infrastructure as Code from Scratch
#30Did that guy just suggest that to make infrastructure-as-code easier to understand we should make it more like CSS ?
What i think we really need: 1. A low-level, open-ended language for describing infrastructure; it should have absolutely no facilities for abstraction, should be human-legible and machine-readable (so based on JSON, probably), and should be applicable to everything from configuring physical hosts and switches up to containers. 2. For each kind of infrastructure, a tool which can apply that language to the infrastruc…