Terraform Config Root Setups

I’ve seen things you wouldn't believe.

Python deterministically generating terraform HCL files based on yaml.

Execution wrappers that encapsulate terraform in CI/CD to parse the json output and prevent database deletion, but apply everything else.

Scripts that pull every git repo and execute every terraform file they can find while walking the directory tree.

Terraform is about 80% of the way to a good tool, that last 20% is a ball-ache and solved totally differently every time; the best setups I’ve seen is where terraform just “hands off” to something else after making a minimum infrastructure.

But, otherwise, it can get incredibly messy.

This is terrible. All options seem to assume state is stored locally (not, say, in S3). Many options are just the same as others but only one environment, or with some project-specific difference like 'there is a backend and a frontend' or 'a few services' which has nothing to do with how you structure the terraform side of it.

All of them either don't address or use multiple directories to handle multiple environments (and assume a static list). What?! No, use terraform workspaces, or a single module that you instantiate for each environment. Or terragrunt if you really want. TFA is just a blogspam mess from either AI or someone that's spent about 20 minutes with a Youtube video to learn terraform & push out their latest deep dive content guys.

The client I'm contracted to is all-in on Terraform Cloud. (TFC)

TFC uses workspaces, which annoyingly aren't the same thing as terraform workspaces. I've divided up our workspaces into dev, qa, staging, and prod, and each group of workspaces has the OIDC setup to allow management of a specific cloud account. So dev workspaces can only access the dev account, etc etc. Each grouping of workspaces also has a specific role that can access them. Each role then has its own API key.

The issues I've run into are mostly management of workspace variables. So now I have a manager repo and matching workspace that controls all the vars for the couple hundred TFC workspaces. I use a TFC group API key for the terraform enterprise provider, one provider per group. This prevents potential mistakes where dev vars could get written to qa, etc etc.

  Manager repo
  - dev TFE provider
  - qa TFE provider
  - staging TFE provider
  - prod TFE provider

Workspace variables are set by a single directory of terraform, so there's good sharing of the data and locals blocks.

I use lists of workspaces categorized by "pipeline deployers" and "application resource deployers", along with lists of dev, qa, staging, and prod workspaces. I then use terraform's "setintersection" function to give me "dev pipeline" workspaces, "prod app" workspaces, etc. I also do the same with groups of variables, as there's some that are specific to pipeline workspaces, and so on. It works well, and it's nice to have an almost 100% terraform control of vars and workspaces.

I split app and pipeline workspaces based on historical decisions, I'm not sure if I'd replicate that on a new project. The workflow there is that an app workspace creates the resources for a given deployment, then saves pertinent details to a couple of parameters. The pipeline workspace then pulls those parameters and uses them to create a pipeline that builds and deploys the code.

Unfortunately I can't share code from this particular setup, but I do intend to write about it "someday".

Please, please, please use Terraform workspaces: One workspace per environment.

For environment specific things use conditionals:

  nodes = terraform.workspace == "prod" ? 2 : 1

Instead of env conditionals, I strongly recommended feature/functionality conditionals or variables. E.g. var.create_s3 is better than var env == 'prod'