Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Terraform should have remained stateless

bejarano.io

Post reply on HN
Print view View original

251–260 of 329 posts

Re: Terraform should have remained stateless

#251

Post by mountainriver »

enobrev wrote: post #241
stingraycharles wrote: post #149

Earlier quoted context omitted.

The whole idea of the article is that “previous state” is something that can be derived from tags / metadata in the provider’s resources, rather than a standalone file. In other words, this metadata then is the state. Could you give an example of a situation / transition that cannot be captured correctly by managing the state using these types of tags?

That's still storing state, except instead of keeping it within a single file which you have full control over, you're instead sprinkling that state all over your infrastructure and then hoping it doesn't get mangled in between invocations. And also hoping that service's API has all the tools needed to find your scattered state within a reasonable amount of time in order to diff any changes you make in your declarati…

Yeah but then you don’t have to worry about the state getting out of sync which is one of the biggest problems. Also versioning is a pain

Re: Terraform should have remained stateless

#252

Post by mountainriver »

pharmakom wrote: post #98

The article misses a key bit of information TF needs when making a plan: 4. The previous Terraform configuration This is effectively stored by state. We need this because if a resource is removed from the new config then Terraform needs to be able to delete the existing resource from the world. If we don’t have the state then Terraform must either: 1. Not delete it from the world 2. Or risk deleting something not man…

I think the trade offs of a stateless terraform would be favored by many

Re: Terraform should have remained stateless

#253

Post by nathants »

jen20 wrote: post #232
nathants wrote: post #70

Earlier quoted context omitted.

using names to uniquely identify infrastructure is very convenient and how most of aws already works. why not take advantage of that instead mapping name to uuid via state? in what scenario are non-stable infrastructure names useful?

It certainly isn’t how “most” of AWS works. Nothing makes the name tags for important APIs like VPC or EC2 even exist, yet alone be unique. They use provider-assigned IDs instead.

vpc and ec2 are the big ones. they both support tagging atomically during create though, so it’s fine?

for ec2 i’m not sure i’d want unique names per instance. typically i prefer groups of instances with the same name.

Re: Terraform should have remained stateless

#254

Post by nathants »

taeric wrote: post #221
nathants wrote: post #70

Earlier quoted context omitted.

using names to uniquely identify infrastructure is very convenient and how most of aws already works. why not take advantage of that instead mapping name to uuid via state? in what scenario are non-stable infrastructure names useful?

If you are using cloud formation, it already does a uuid like name on resources it creates.

i’m not. i do my own thing[1] with the go sdk.

1. https://github.com/nathants/libaws

Re: Terraform should have remained stateless

#255

Post by ImPostingOnHN »

mountainriver wrote: post #251
enobrev wrote: post #241

Earlier quoted context omitted.

That's still storing state, except instead of keeping it within a single file which you have full control over, you're instead sprinkling that state all over your infrastructure and then hoping it doesn't get mangled in between invocations. And also hoping that service's API has all the tools needed to find your scattered state within a reasonable amount of time in order to diff any changes you make in your declarati…

Yeah but then you don’t have to worry about the state getting out of sync which is one of the biggest problems. Also versioning is a pain

yeah, but then you DO have to worry about scanning every single resource you control, every time you plan and run

Re: Terraform should have remained stateless

#256

Post by hellcow »

ImPostingOnHN wrote: post #255
mountainriver wrote: post #251

Earlier quoted context omitted.

Yeah but then you don’t have to worry about the state getting out of sync which is one of the biggest problems. Also versioning is a pain

yeah, but then you DO have to worry about scanning every single resource you control, every time you plan and run

I built pretty much the system you’re describing for my company (a stateless terraform alternative) and this scan that you cite as a negative happens in… 100ms in parallel? Roughly the same amount of time it would take to download a state file? Dunno about you but ensuring state is always accurate and in-sync is well worth the trade-off to me.

Re: Terraform should have remained stateless

#257

Post by erikerikson »

ldoughty wrote: post #15

Basically agree with the article. I've used direct cloud formation, AWS SAM, Ansible, terraform, and AWS CDK to spin up infrastructure... My hard line opinion is that if something NEEDS state management to exist and update, it's a pet, treat it like a pet. Don't mix pets with the rest of your automated machinery except to the minimum extent required, when absolutely necessary. We had to rewrite an Ansible role becaus…

Right here with you on this, I can add ARM templates (in complete mode) to the list.

I might suggest the Serverless framework (no serverless needed) which lets you write CFTs in most formats you prefer, use variables (including pulling config from S3), provides the canned scripts you otherwise add, and so on.

I wish MS would unbreak it's plugin to Serverless to support full ARM.

As others have written, there is state in the CFT/ARM/etc. services via named deploys. As you note, that allows for the deletion of removed things (including things for which you've specified another name). That's great.

The challenge with Terraform state use is that it is often too tetchy about what it doesn't know about and often reacts by tearing down and rebuilding assets, causing unavailability and data loss. It was even observed doing this for an entire stack due to a transitory network failure at PayPal. I loved --auto-approve because I don't want to reintroduce human failure and friction into the mix but Terraform's models and dependency on "community or when we get to it" means it just isn't safe the way the native provisioning services are. As you note, alignment of incentives and all.

Re: Terraform should have remained stateless

#258

Post by ImPostingOnHN »

Jasp3r wrote: post #164
janosdebugs wrote: post #11

I respectfully disagree. If TF was stateless, you'd have to manage situations by hand that involve changing the idempotency key, such as the name of a VM. You'd also have to manage situations by hand where a resource is removed from the config. The whole point of TF is that it has state and doesn't require workarounds for these scenarios. Yes, you have to maintain state, but the state problems usually come from buggy…

Why would renaming be impossible to solve by a stateless terraform? In a cloud setup with 3 VMs, and you rename 1 VM from A to B: Measured cloud setup would have VMs A + X + Y New cloud setup would specify VMs B + X + Y You can easily identity X and Y as their names go unchanged, A and B would have similar config/metadata, instead of assuming A would be added and B would be removed you can ask the user if a rename ha…

as a user, I don't want to be asked to interactively guide what is supposed to be automated as its entire raison d'etre

Re: Terraform should have remained stateless

#259

Post by erikerikson »

takeda wrote: post #55
tokamak wrote: post #48

AWS should really find a way to replace CF with something sane like TF so we can have both cloud resources and cloud workload described with native and managed tech. TF is pain but necessary.

I think your perception might be due to using TF. To me, once CF added YAML support it's much more straight forward to use.

Use the Serverless framework for your cFT (no serverless needed) to add in richer variables (e.g. config find fetching) capability too.

Re: Terraform should have remained stateless

#260

Post by cppforlife »

pst wrote: post #67

I maintain a Terraform provider for Kubernetes. And one of the main reasons for that is because the Terraform state ensures purging of deleted resources. Something that kubectl is not capable of. The lastAppliedConfig annotation does not help for purging, because once the manifest has been deleted on disk, there is no way of knowing what to delete from the server. The unusable apply --purge flag is the best example o…

i think kubernetes is not a great example in favor of more client state (like tf) since k8s has uniform resource structure (metadata.*) and first class labeling support. but as you point out kubectl doesnt use labels well (at least imho).

when building https://carvel.dev/kapp (which i think of as "optimized terraform" for k8s) the goal was absolutely to take advantage of those k8s features. we ended up providing two capabilities: direct label (more advanced) and "app name" (more user friendly). from impl standpoint, difference is how much state is maintained.

"kapp deploy -a label:x=y -f ..." allows user to specify label that is applied to all deployed resources and is also used for querying k8s to determine whats out there under given label. invocation is completely stateless since burden of keeping/providing state (in this case the label x=y) is shifted to the user. downside of course is that all apis within k8s need to be iterated over. (side note, fun features like "kapp delete -a label:!x" are free thanks to k8s querying).

"kapp deploy -a my-app -f ..." gives user ability to associate name with uniquely auto-generated label. this case is more stateful than previous but again only label needs to be saved (we use ConfigMap to store that label). if this state is lost, one has to only recover generated label.

imho k8s api structure enables focused tools like kapp to be much much simpler than more generic tool like terraform. as much as i'd like for terraform to keep less state, i totally appreciate its needs to support lowest common denominator feature set.

common discussion topics:

* whats the lowest common denominator for apis that need to be supported

* how much state to store client side vs server side (in the api itself e.g. tags or in "assistive service" e.g. s3 api)

* is it enough to just store resource identifiers vs whole resource content (e.g. can resource content be retrieved at a later point; if content is stored, is it sensitive)

* how easy is it to recover from complete state loss

Post reply on HN