Rethinking Infrastructure as Code from Scratch

> Sorry, but if it's doesn't have abstraction it's not readable.

This is why i very carefully wrote "legible" instead of "readable".

> You get smothered in details. No forest, just trees. People will just create a framework that generates the file like Sass does with CSS.

That ... is exactly what i was suggesting?!

My thoughts have been going into another direction entirely:

- We need to get rid of YAML. Not only because it's a horrible file format but also because it lacks proper variables, proper type safety, proper imports, proper anything. To this day, usage & declaration search in YAML-defined infrastructure still often amounts to a repo-wide string search. Why are we putting up with this?

- The purely declarative approach to infrastructure feels wrong. For instance, if you've ever had to work on Gitlab pipelines, chances are that already on day 1 you started banging your head against the wall because you realized that what you wanted to implement is not possible currently – at least not without jumping through a ton of hoops –, and there's already an open ticket from 2020 in Gitlab's issue tracker. I used to think, how could the Gitlab devs possibly forget to think of that one really obvious use case?! But I've come to realize that it's not really their fault: If you create any declarative language, you as the language creator will have to define what all those declarations are supposed to mean and what the machine is supposed to do when it encounters them. Behind every declaration lies a piece of imperative code. Unfortunately, this means you'll need to think of all potential use cases of your language and your declarations, including combinations and permutations thereof. (There's a reason why it's taken so long for CSS to solve even the most basic use cases.) Meanwhile, imperative languages simply let the user decide what they want. They are much more flexible and powerful. I realize I'm not saying anything new here but it often feels like as if DevOps people have forgotten about the benefits of high-level programming languages. Now this is not to say we should start defining all our infrastructure in Java but let's at least allow for a little bit of imperativeness and expressiveness!

Author here. This is correct. I have used Pulumi and I love CDK. I think the models in Pulumi and CDK are both great, but they tend to be either too low level or too high level.

For example Pulumi has a `awsx.ecs.FargateService` class that can deploy a container to AWS Fargate, but the API is limited and you don't really have the capability to remix it into other scenarios. For example you have the capability to attach a load balancer to a service as ingress, but what if you want to attach an API Gateway, or just setup a Route53 record for direct public IP ingress?

CDK has a few more variations in what you can do but it tends to be even worse because it has class names like: `ApplicationLoadBalancedEc2Service` and `NetworkLoadBalancedFargateService`. Switching your infrastructure requires switching to an entirely different top level when you define your infrastructure.

What I am proposing with the extensions interface is that we build more a class inheritance style model, kind of like CSS, to let people mix in alternative behaviors, alternate configurations, or even attached resources in an additive way, onto the existing infrastructure as code model.

In the example prototype on Github you can see how this works: https://github.com/cdklabs/cdk-ecs-service-extensions

All types of mutations, from configuration mutations, to attached resources are just extensions that you can use the `ServiceDescription.add(Extension)` method to attach to your existing infrastructure, kind of like just layering on another CSS class onto your DOM element. I think for large and complex infrastructure this will be a more scalable, decouplable, and readable approach.

A lot of people will argue that state helps protect against drift, but the real reason I find that you have to have state is to store values that won't be returned a second time and still construct and connect the graph of resources in the IaC templates. For example, if you declare the need for an RDS database and connect its output credentials into another application, you'll need state in order for the applies to work a second time because you'll never be able to retrieve the values from the target provider again.

I think GitHub Actions got it right. Mostly declarative workflow, but with reusable Actions to extend the DSL.