On a related note, CDK for Terraform allows DevOps practitioners to use a variety of programming languages instead of HCL. I've really enjoyed modeling my AWS environments with Python using Terraform only as the engine. More info here: https://github.com/hashicorp/terraform-cdk
>"On a related note, CDK for Terraform allows DevOps practitioners to use a variety of programming languages instead of HCL." Out of curiosity what is it that people generally don't like about HCL?
Terraform 0.15 General Availability
131–140 of 239 posts
Re: Terraform 0.15 General Availability
#132Earlier quoted context omitted.
Have you ever set something up manually via AWS console, then 6 months later totally forgot the steps you took and end up wasting a lot of time reverse engineering what you did in order to make a comparatively small change? After that perhaps you vowed to take better notes, so the next time you do it that way, but then 6 months later you find that you missed some detail, or there was some changes in between that were…
The jump from what you mentioned in paragraph 2 to 3 is not necessarily Terraform. You can use other tools, like Ansible, that IMO have a much better framework than Terraform. I use Terraform for extremely simple stuff that is easy to destroy/recreate. Projects of bigger scale IMO are better served with Ansible and friends.
You need to write idempotent ansible from the start. No excuses.
Then you would implement a retry logic in ansible since your cloud APIs will fail for weird reasons or you really must wait that a resource in AWS is truly ready before you run another task. ( for example vpn gateway in AWS must be ready and then you can attach a vpn connection, or a route53 zone record has to propagate before you do something with it. But in the meantime your other tasks can continue in parallel. Only the vpn tasks needs to wait)
Then after a while with all your resources created with your idempotency implemented in ansible, your code will roughly first check ( with timeouts, retries) if you actually can skip tasks since there is nothing to do. The checks increase, your playbook takes longer and longer to execute. Maybe you try to parallelize it in some way. In the meantime a colleague of yours executes the same playbook in their machine or pipelines, and now you see weird side effects. Maybe you start implementing some locking behavior.
At this point we did not do any code reviews yet and we did not answer the question: “how should the infrastructure look like?”
This especially means: how do you actually delete resources with Ansible? So you Start to introduce a kind of “state” for your resource.
So, if you squint hard enough, idempotent, resilient, parallelizable, thread-safe, fast ansible with a state is what terraform solves. If you look at tools like terratest you even get to do unit/integration testing for your infrastructure code.
So as soon as you are more than 1 person handling infrastructure following best practices like code review and testing, without getting insane, use terraform.
If you are alone and don’t want to follow the above best practices, you still have to be very good in writing idempotent, resilient ansible code.
My hypothesis is that the intersection of people who write high quality (idempotent, resilient) ansible code but people who do not care about code review/testing, is quite small.
Once you do not work in isolation any more, either in a team or as a consultant who needs to hand over their work (and Teach people how idempotent ansible works), I would favor terraform any time.
You could even call ansible from terraform if you need some ansible integration, where ansible has a nicer api for you. But let terraform handle the retries, state management and ansible could just be the executing part. So ansible is more like a fancy bash script/function which you call in an orderly manner.
Re: Terraform 0.15 General Availability
#133Earlier quoted context omitted.
Are all variables, conditionals, templating, and loops done in Python? Or is some of that still needed on the Terraform side?
Basically you create the desired state DAG in procedural code, rather than the TF DSL. Blithe diffing and applying are the same.
Re: Terraform 0.15 General Availability
#134Earlier quoted context omitted.
How do they differ?
+1 to Pulumi. xyzzy123 already described the differences between Pulumi and terraform, but I want to add one key way in which they are similar: Pulumi uses terraform under the hood. We get all of the reliability of terraform, but with a much more powerful runtime engine.
Re: Terraform 0.15 General Availability
#135Re: Terraform 0.15 General Availability
#136Hijacking a bit, but does anyone have any good resources/guides around managing terraform state in larger organizations? Terraform enterprise seems to address this but I was wondering if there's workflows that allowed subsections of infrastructure (think teams or systems) and didn't rely on a re-evaluation of the entire organization's assets. So far the only approach I've seen is having protected high level (VPC, sub…
Re: Terraform 0.15 General Availability
#137Hijacking a bit, but does anyone have any good resources/guides around managing terraform state in larger organizations? Terraform enterprise seems to address this but I was wondering if there's workflows that allowed subsections of infrastructure (think teams or systems) and didn't rely on a re-evaluation of the entire organization's assets. So far the only approach I've seen is having protected high level (VPC, sub…
Some core components that most will include as remote state, and you can discover and use as remote state stuff from other teams, too.
Quite basic, but something tells me this could very well be intentional, so that the enterprise offering has a clearly increased value.
Re: Terraform 0.15 General Availability
#138Hijacking a bit, but does anyone have any good resources/guides around managing terraform state in larger organizations? Terraform enterprise seems to address this but I was wondering if there's workflows that allowed subsections of infrastructure (think teams or systems) and didn't rely on a re-evaluation of the entire organization's assets. So far the only approach I've seen is having protected high level (VPC, sub…
This way, the per account setup (represented as a repo) is relatively small, common patterns are standardized, and there is still room for experimentation.
Re: Terraform 0.15 General Availability
#139Earlier quoted context omitted.
I think there's a middle ground if you're not sure how to fix a mistake in Terraform but you know how to do it in the console: * Make your changes by hand * Right afterward, run "terraform plan" to see how Terraform would undo your changes * Edit your Terraform config to reflect those changes, and run "terraform plan" again to make sure you caught everything. Repeat until it's a no-op. Now you've got a log of what yo…
You can do it that way, but I find the tf docs easier and more concise to use than clicking around the AWS UI.
Re: Terraform 0.15 General Availability
#140Earlier quoted context omitted.
The jump from what you mentioned in paragraph 2 to 3 is not necessarily Terraform. You can use other tools, like Ansible, that IMO have a much better framework than Terraform. I use Terraform for extremely simple stuff that is easy to destroy/recreate. Projects of bigger scale IMO are better served with Ansible and friends.
Ansible looks fine when you start, but gets painful. You need to write idempotent ansible from the start. No excuses. Then you would implement a retry logic in ansible since your cloud APIs will fail for weird reasons or you really must wait that a resource in AWS is truly ready before you run another task. ( for example vpn gateway in AWS must be ready and then you can attach a vpn connection, or a route53 zone reco…