Terraform should have remained stateless • Hacker News Board

Re: Terraform should have remained stateless

#211

Post by oneplane » Sat, May 28, 2022, 12:54 PM UTC

Well, this would ruin the point of terraform and turn it into Ansible... You really need all three to get any use out of terraform:

  - The desired state in your .tf files
  - The actual state in your provider (what you describe)
  - The expected state in the state file

This essentially gives you drift detection and delta updates. A simple "terraform refresh" and "terraform plan" read your actual state and does a diff between desired state and actual state. If all you had was the real world state, then the absence of resources gives you zero information. The other way around: planning a change without comparing it against a stored state gives you no way to actually purge the real world state.

You could technically argue that everyone should keep their .tf files in VCS/SCM and then have terraform first check the real world against the previous commit before creating a delta based on the current changes, but then you're just moving state to Git which is already a state backend...

The triangle this creates is why terraform generally is better than most other IaC systems which either don't have all three legs (and thus collapses them into a 2-dimensional also-ran tool) or they do but only for one special system (i.e. only AWS/GCP/Azure and no integration with anything else).

Next thing you know someone is coming to advocate against locking and hash comparison...

Edit: the best 'simple' explanation I could come up with is: you can't remove or update what you don't know shouldn't exist anymore. And you can't realistically 'download' the configuration of an entire cloud to 'check' all the tags for state information.

Re: Terraform should have remained stateless

#212

Post by usrn » Sat, May 28, 2022, 12:59 PM UTC

kkdaemas wrote: post #191

usrn wrote: post #172

Earlier quoted context omitted.

>- The risk of losing state information by a resource being deleted outside of Terraform is greater This isn't losing state information though! That is the state. If state information were kept outside this it would now be wrong which means terraform *would do the wrong thing.

No because the metadata on the deleted resource is now lost. With the information being stored outside the resource, we know that it was deleted and the metadata about it.

That's not state information though, that's metadata.

Re: Terraform should have remained stateless

#213

Post by kubanczyk » Sat, May 28, 2022, 1:02 PM UTC

pixelcort wrote: post #104

Can the git repo that the terraform config lives inside of serve as state? For example, one commit ago this part of config was there and now it’s gone, showing temporal intent that it should be deleted in real life as well?

I've played with your idea, hoping to easily demolish it. It is not bullet-proof at all, but I declare it brilliant. It uses simple tools and it does work around the cloud's inability to efficiently query all possible types of resources in existence.

Re: Terraform should have remained stateless

#214

Post by pst » Sat, May 28, 2022, 1:03 PM UTC

GeneralMayhem wrote: post #77

pst wrote: post #72

Earlier quoted context omitted.

The delete this approach is imperative though. If you aim to be declarative you need a way for the tool to be able to determine actions necessary to go from current to new desired configuration. You need to store previous applied config somewhere, to be able to determine if something needs purging in a declarative way.

You don't need to store the full old configuration anywhere other than as part of the current configuration. All you need is a list of IDs that existed in previous configurations. Something like: current_tables { TableA { Column1[string] Column2[bool] } } removed_tables: ["TableOld", "AnotherOldTable", ...] Depending on your ergonomic preferences, you could also accomplish that by keeping the old table configs and ad…

Yes, but setting up and handling edge cases of Terraform state causes the effort. Once you have it, storing just IDs or more doesn't make a difference anymore.

Re: Terraform should have remained stateless

#215

Post by pst » Sat, May 28, 2022, 1:04 PM UTC

silverlyra wrote: post #81

pst wrote: post #72

Earlier quoted context omitted.

The delete this approach is imperative though. If you aim to be declarative you need a way for the tool to be able to determine actions necessary to go from current to new desired configuration. You need to store previous applied config somewhere, to be able to determine if something needs purging in a declarative way.

It sounds to me more like Puppet's "ensure absent"; still declarative in the sense that you can keep it around and it will continue to clean up any zombie instances that recur. And this is only during incremental adoption, where you'd soft-delete resources in your config by switching them to tombstones instead of removing them entirely, and adding tombstones for legacy unmanaged resources you want to remove (which bu…

Keeping the tombstones around in the configuration I have to maintain instead of the state the tool maintains for me also increases the effort for me though. So either I have the effort of setting up the remote state and handling some edge cases. Or handling the shortcomings of stateless in my own code base.

Re: Terraform should have remained stateless

#216

Post by [deleted] » Sat, May 28, 2022, 1:05 PM UTC

ntoskrnl wrote: post #168

Jasp3r wrote: post #163

Earlier quoted context omitted.

That cloud providers provide such muddy/inadequate APIs that it is impossible to view the state they are in is a very bold claim to make. Do you have an example to back this up?

No DNS provider I've ever used has a way to store metadata alongside a DNS record (like an individual A record)

[deleted]

Re: Terraform should have remained stateless

#217

Post by pst » Sat, May 28, 2022, 1:10 PM UTC

phrotoma wrote: post #142

pst wrote: post #67

I maintain a Terraform provider for Kubernetes. And one of the main reasons for that is because the Terraform state ensures purging of deleted resources. Something that kubectl is not capable of. The lastAppliedConfig annotation does not help for purging, because once the manifest has been deleted on disk, there is no way of knowing what to delete from the server. The unusable apply --purge flag is the best example o…

Could you elaborate on the poor usability of the --purge flag?

It's not trivial to get the labels correct to avoid collateral deletions. Also, while it makes sense, I and many teams I consulted with found it rather unintuitive that apply --purge with a label selector will also only update resources with the label. Not all resources that are in the list of resources. Last time I checked it was also still marked experimental and has been for years.

Re: Terraform should have remained stateless

#218

Post by politician » Sat, May 28, 2022, 1:12 PM UTC

rtp4me wrote: post #173

rwiggins wrote: post #12

Having worked with both Terraform and Ansible code that created AWS resources - and operates very similarly to the model described here [0, see `filters` arg] - I generally disagree. For example, if you changed an ID in your stateless Terraform, you'd have to insert some kind of code to destroy (or rename, if possible) the old resource. Or modify the Terraform DSL to include that kind of information, I suppose, and k…

I am starting on a journey to deploy resources in the big three cloud providers (AWS, Azure, Google) and could use your input. So far, we have been working with Ansible to provision our AWS resources. Because we find Ansible to be a pain to use, we are considering Terraform - especially when working with multiple cloud providers. Do you have any advice or best practices to properly deploy and manage instances using T…

Terraform is a configuration language over the top of providers that expose their own abstractions.

The first thing to realize about using Terraform is that _you_ can extend it with your own providers written in a language like Go and _you_ can cobble together your own modules written in the TF configuration language to orchestrate multiple providers or do repeatable work. There are really solid open-source modules for AWS operations that smooth out the kinks in the AWS API.

Second, use state and check it into an S3/R2 bucket. Keep your TF scripts in Git, and check them in too after changes. Make a checklist of what steps you take each time you modify a resource (first in the script, then in the state/live).

Third, learn the command line tools used to fix horked state deployments. It'll happen from time to time, and there are GOOD tools that already exist to fix issues. Also, the state is JSON, and you _can_ edit it by hand if you need to get something to work.

Remote cloud provider configuration is a complex problem space akin to programming-at-a-distance. It's hard because APIs are trash, APIs go down or flap in the middle of action, and APIs are slow, so debugging is tricky. Early on, a full tear down and rebuild policy helps, but quickly the slow APIs/slow cloud operations make you more reluctant to start from scratch.

Oh, and databases require a completely different management approach.

That said, I still endorse Terraform over Pulumi or the AWS CDK.

Re: Terraform should have remained stateless

#219

Post by pst » Sat, May 28, 2022, 1:14 PM UTC

anticristi wrote: post #148

pst wrote: post #67

I maintain a Terraform provider for Kubernetes. And one of the main reasons for that is because the Terraform state ensures purging of deleted resources. Something that kubectl is not capable of. The lastAppliedConfig annotation does not help for purging, because once the manifest has been deleted on disk, there is no way of knowing what to delete from the server. The unusable apply --purge flag is the best example o…

> I think the state mainly exist to know what has been created in the past but since been deleted from manifests and therefore needs to be purged. The caching/performance argument is rather weak, because Terraform refreshes by default anyway before any operation. Beautiful summary. For resources with flexible tags, one could easily imagine tags like Kubernetes's: terraform.io/name terraform.io/instance However, for t…

Flux, IIRC, uses labels or annotations to do purging. Helm I'd argue falls into the state category with the secrets if uses to track releases.

I do everything with Terraform so I'm not super familiar with either of them. But teams are free to choose their poison.

Re: Terraform should have remained stateless

#220

Post by pst » Sat, May 28, 2022, 1:27 PM UTC

boesboes wrote: post #155

Personally I think the custom dsl’s are a bigger issue. I spend a lot of time wrangling tf to have reusable, configurable modules. The more i use tf, the more i think it would be better to remove _all_ dynamic features and use a real language to generate tf configs as flat, static files.

Yes, conditionals and loops in TF are limited and have various annoying and surprising edge cases. But if something is hard to do with the Terraform DSL it's usually a good idea to reconsider if it is really something that one should be doing.

We want infrastructure automation to be boring and just work.

The risk with general purpose programming languages is that people will always find a way to outsmart themselves. Yes, sure, you can use the testing tool chain of the language of your choosing. But it's not like we have figured out to write software without bugs, despite all the awesomeness of modern languages.