Live data from Hacker News

Rethinking Infrastructure as Code from Scratch

nathanpeck.com

71–80 of 124 posts

Re: Rethinking Infrastructure as Code from Scratch

#71
post #48
post #7

Earlier quoted context omitted.

What i think we really need: 1. A low-level, open-ended language for describing infrastructure; it should have absolutely no facilities for abstraction, should be human-legible and machine-readable (so based on JSON, probably), and should be applicable to everything from configuring physical hosts and switches up to containers. 2. For each kind of infrastructure, a tool which can apply that language to the infrastruc…

> 1. A low-level, open-ended language for describing infrastructure; it should have absolutely no facilities for abstraction, should be human-legible and machine-readable (so based on JSON, probably), and should be applicable to everything from configuring physical hosts and switches up to containers. Sorry, but if it's doesn't have abstraction it's not readable. You get smothered in details. No forest, just trees. P…

I think what is considered low level has to be qualified here. In my opinion I should be able to request virtual cores and memory the same way I do in an os. When in c I can malloc, calloc, or mmap or brk if I'm on a Linux system. If I'm doing concurrent thing in c I have to specifically request a thread then manage it. If I can do these, why can't I have a library where I can make those requests to my cloud provider from my code?

To me, low level would be individually requesting resources as I need them with abstractions that let me ignore physical vs virtual distinctions. To me, this would be fairly readable. C like code requesting then allocating memory and threads just makes sense to me and I don't really care about if they're part of a larger machine. But I also think of low level from my code's perspective not from my infrastructure's perspective.

I'm curious what you think would need to be targeted for something to be both cloud native and low level? I rarely hear people discuss this type of classification.

Re: Rethinking Infrastructure as Code from Scratch

#72
post #67

Earlier quoted context omitted.

> “there’s no way it has to be like this.” It really doesn't, but alas: worship at the altar of unnecessary cloud complexity or be cast out to the on-prem Elysian field.

I think you've muddled "Elysian fields" (the reward of heroes) with "being put out to pasture" (superannuation).

It's both :)

Re: Rethinking Infrastructure as Code from Scratch

#73
post #5

Did that guy just suggest that to make infrastructure-as-code easier to understand we should make it more like CSS ?

I think if you use CSS as a source of lessons learned after the mistakes were implemented and cemented, then yes. If you aim for where CSS was intended, and make decisions which do not compromise that direction, then it's fine goal.

The layering of distinctly-defined concerns contained in separate files which collectively project a merged specification to an IaC tool is a good idea, I think.

Re: Rethinking Infrastructure as Code from Scratch

#74
post #21

Earlier quoted context omitted.

Adding a "real" programming language makes certain things easier, such as abstraction, but IMO they are too powerful for the task at hand. Do we really want an infrastructure description to be able to execute arbitrary code?

Yes, that’s exactly what we want. Things like Terraform also permit this via provisioners, and CloudFormation permits it via execution of lambda functions. Almost any non trivial infrastructure requires it.

With Terraform you can statically analyze the infrastructure definition with some guarantees of determinism etc. Arbitrary execution is allowed, as you say, but only in well-contained places, such as local_exec.

How can this work if, say, TypeScript is used as the definition language?

Re: Rethinking Infrastructure as Code from Scratch

#75
Is this a trick to make infrastructure/devops engineers learn TypeScript?

But, hey, when looking at the origins of OOP and its main uses (back then: Simulations and UI): Maybe this is exactly what one needs to describe and setup infrastructure and there have been various projects going in that direction.

Make message passing truly async, throw in Garbage Collection, make it dynamic (i.e. creating a new instance of an object leads to some sort of deployment) and voila: Your traversable, introspect-able object graph is now a representation of your infrastructure.

Re: Rethinking Infrastructure as Code from Scratch

#76
post #69

Earlier quoted context omitted.

This does not work. Say I have resource A with property X=1 I define in IAC. Someone comes along and modifies X=2 outside of state. With your way, the IAC tool would see that change and think it was naturally part of the desired state, whereas stored state will catch the drift. And before anyone says “well dont modify outside of IAC” I say 1) that’s often impractical and 2) sometimes automation can modify resources o…

First, you can guard against it. Just periodically re-run the infra code in the "dry-run" mode (from a CI/CD system) and scream if you see any differences. Second, this is still fine. Don't make changes outside of the IAC control. And if you do make them, retro-fix the IAC files until there is no diff with the actual state. Third, IAC should have an option to ignore some changes. > Also, dynamically creating state cr…

> Second, this is still fine. Don't make changes outside of the IAC control. And if you do make them, retro-fix the IAC files until there is no diff with the actual state.

This doesn't work in practice. Some aspects of the business want to tweak things and it should be reasonably guaranteed that the automated side never touches it.

Terraform state gives this assurance because it won't destroy resources not under its state.

Re: Rethinking Infrastructure as Code from Scratch

#77

this is great, but I would argue the biggest issue with infrastructure as code is this: the structure and syntax for AWS is entirely different from Azure is entirely different from GCP. Instead of abstracting to CSS, I would argue modeling what Bytecode did in java for multi-operating system, we should do for infrastructure of code. That way, you could easily replicate in different environments, free yourself from ve…

This is what Multy (https://github.com/multycloud/multy) aims to do

Re: Rethinking Infrastructure as Code from Scratch

#78
post #3

What the author appears to miss is that many existing IAC tools permit exactly this. CDK, CDKTF and Pulumi all use general purpose programming languages, so reusing parameter objects in the way that is described is trivial - indeed it is so close to second nature that I would not even think to write it down. Indeed, it's not uncommon to share functions that make such parameter objects via libraries in the package eco…

I'm the author of the article. I actually shared a prototype towards the end of the article, of the idea implemented in CDK. I agree that you can do a lot of this already in CDK, CDKTF, and Pulumi. I just don't think most people are actually doing it (yet).

I've been using CDK since early beta, and have actively contributed to the project. But most people that I'm seeing using it today are just wrapping up new higher abstractions with a simpler but limited API. I think that is an okay start, but I want to encourage people to think more about the infrastructure as being made up of traits/classes/adjectives that are mixed together to form the final product. The same way we have class inheritance in object oriented programming, or CSS classes in HTML.

Eventually the dream is to be able to provide a library of standard infrastructure as code mix-ins that can be applied to your cloud architecture. For example imagine if you could apply a generic "Graviton" trait to a CloudFormation or Terraform or Pulumi or CDK stack and it would automatically configure the appropriate properties on your EC2 instances, and your RDS database, and your Fargate tasks, and all your other compute. With CDK's built-in container and image builds it could even run your Dockerfile based build inside of the matching architecture as well, all based on a single trait that you add to your stack.

There are a wide variety of these types of "traits" that you might be able to build and add.

Re: Rethinking Infrastructure as Code from Scratch

#79

I don't think the author has tried Pulumi, which can do exactly this kind of thing.

He's suggesting something closer to Pulumi than a declarative (Cloudformation, Terraform), but with more of an inheritance model to apply blanket attributes to the targeted resources. This is possible with Pulumi but requires a lot of boilerplate and some monkeypatching.

Well, Pulumi is declarative, so I don’t really understand where this is coming from

Re: Rethinking Infrastructure as Code from Scratch

#80
post #5

Did that guy just suggest that to make infrastructure-as-code easier to understand we should make it more like CSS ?

I think the part that upsets me is he specifically calls out the problem CSS was meant to fix, but then presents the current usage of CSS doesn't fix the problem but instead inverts it. Instead of having to scour code for instances of an attribute you wish to change, you're scouring output for potential negative consequences to a 1px margin shift you added to a style used everywhere. It seems like this will just ampl…

Author of the article here. I totally agree with you on this. There are definitely tradeoffs. However I think the risk of this approach is arguably better than the risk of having every team implementing their own IaC from scratch.

With the CSS example you can have a bunch of very junior developers using Bootstrap and Tailwind with minimal knowledge of CSS and get great results precisely because they don't actually have to change the CSS classes inside of the CSS framework. Junior devs don't make "1px margin shifts" as often because the framework has good margins out of the box. Additionally if needed any "1px margin shifts" that they do make could be enforced by linter or other code policies to happen as a new class, with visibility and limited impact, rather than a change to an existing underlying CSS framework class.

The same could be true for our infrastructure as code if we had similar prepackaged configuration mix-ins. It actually lowers the risk of junior developers making "1px margin shifts" because the IaC framework has sensible config mix-ins out of the box. Most modern IaC just hands the full API surface area to a junior dev, perhaps provides a few examples, and then hopes that they do the right thing with it.

I think modern frameworks like AWS Cloud Development Kit are doing the right thing by implementing higher level methods like `database.connections.allowFrom(service)`, which automatically configures minimal access security groups with the default port of a stack construct, etc. This prevents juniors from making mistakes like opening up every port to every IP address on the internet. However, I think we need the underlying infrastructure as code language to also gain an understanding of mixin methods and traits that can be reused like that, and it needs to be more general purpose.

The "CSS" analogy is a starting point for introducing people to this idea of building reusable resource configuration bundles and resource mutation methods that can be plugged into multiple infrastructure resources and reused, instead of having every IaC stack be its own special hand crafted thing.

Post reply on HN