Live data from Hacker News

Terraform should have remained stateless

bejarano.io

271–280 of 329 posts

Re: Terraform should have remained stateless

#271
post #67

I maintain a Terraform provider for Kubernetes. And one of the main reasons for that is because the Terraform state ensures purging of deleted resources. Something that kubectl is not capable of. The lastAppliedConfig annotation does not help for purging, because once the manifest has been deleted on disk, there is no way of knowing what to delete from the server. The unusable apply --purge flag is the best example o…

i think kubernetes is not a great example in favor of more client state (like tf) since k8s has uniform resource structure (metadata.*) and first class labeling support. but as you point out kubectl doesnt use labels well (at least imho). when building https://carvel.dev/kapp (which i think of as "optimized terraform" for k8s) the goal was absolutely to take advantage of those k8s features. we ended up providing two…

Was just about to call out kapp but I see Dmitry is on it. We need kapp for all cloud resources

Re: Terraform should have remained stateless

#272
post #69

Earlier quoted context omitted.

Cdk is compiling into CF template. And Pulumi is using Terraform providers under the hood. Writing infra in imperative style can cause ALOT of unseen issues when developers start adding IFology or Design Patterns..

Pulumi also has a newer iteration that uses provider APIs directly. A pain point has been that terraform provides tend to lag significantly with platform APIs, or just miss some functionality entirely.

The providers are maintained by dedicated teams of Amazon, Microsoft, Alphabet, Hashicorp themselves, other big corps.

Pulumi abstractions are updated by a single corpo team.

Ill be EXTREMELY surprised if they wont laaaggg alot more behind when they start supporting more and more platforms.

Its simply a matter of amount of ppl working on the tool.

Re: Terraform should have remained stateless

#273
post #192

Basically agree with the article. I've used direct cloud formation, AWS SAM, Ansible, terraform, and AWS CDK to spin up infrastructure... My hard line opinion is that if something NEEDS state management to exist and update, it's a pet, treat it like a pet. Don't mix pets with the rest of your automated machinery except to the minimum extent required, when absolutely necessary. We had to rewrite an Ansible role becaus…

> I've almost never had this happen with any AWS tool... Sure, there's drift possibilities, but those are controllable by mainly not letting humans do things, and not having multiple cooks in the kitchen changing things in automation, which are good ideas for terraform and Ansible to... This is also what you do with Terraform: that's what the Terraform Cloud product is about (or you can just build a CI pipeline in yo…

> It also sounds like what you're saying is that you can just avoid all these problems by having everything automated from day one, but that's not the reality in any employer I've ever worked for. Unless you're starting a company today and happen to have an experienced infrastructure engineer on staff from day one you're not getting that world

Certainly, it's hard to start it from day one. But it's not that difficult to move into it. If you have buy-in from your developer/operations team. We started with API and CLI calls for our beta version. Our next migration was to use partial Ansible control and we morph that into almost a monolith because of interconnected pieces that were typically required with this design. But it really didn't need to be a monolith, we just wanted to link things together and building a giant monolith was easier to make those references.

So we then split up into smaller Ansible playbooks, and we did lookups to create the linkage, which roughly broke the monolithic pattern and allowed us to do smaller deploys. But we still ran into breaking changes unexpectedly. So we decided to abandon the months of effort we put into Ansible and we started looking at terraform, because one of their salespeople promised our management team that it is cloud agnostic and we would only have to write once. After a week or two looking into that, we realized that we were basically just going to have to remake Ansible modules and we rarely weren't saving anything by migrating. Granted this was two and a half to three years ago. Things might have changed.

We then switch to SAM, and as we did that we extracted the Ansible side out of our deployment and we started redeploying brand new small SAM stacks, and started treating almost everything of our infrastructure as sheep instead of cattle, we completely redeployed our launch configurations our cluster are lambda functions, basically everything except the database, DNS, and the CDN with every deploy. This basically removed state is being an issue because the state is only needed for the first deploy, and we don't technically change the stacks afterwards since we simply replace them every time. For us, this also meant we could easily test and roll back if needed. Since we don't need to change the state of the stack back to its previous state, we simply changed the pointer to the previous stack, which typically was a DNS state change. But like I said, we focused on our states being only around DNS, etc. Which, almost isn't stateful, because our SAM deploys would insert zero weighted DNS records, and The state management is really just adjusting the weighted values.

Re: Terraform should have remained stateless

#274

Earlier quoted context omitted.

Terraform already has to manage those, and yet worse - Changes to the name of objects within the state were entirely manual until 1.0. Changes to objects underneath terraform often break the connection. Terraform doesn't have good tooling built in to find orphan resources, which it would if it were working with stateless objects. Terraform would be much better without state. Not 10x better, but 2x.

I don't quite follow - why would it be able to find orphaned resources if it were working with stateless objects?

Presumably because it would already have all the necessary code to find/enumerate existing resources.

Re: Terraform should have remained stateless

#275

Earlier quoted context omitted.

That's still storing state, except instead of keeping it within a single file which you have full control over, you're instead sprinkling that state all over your infrastructure and then hoping it doesn't get mangled in between invocations. And also hoping that service's API has all the tools needed to find your scattered state within a reasonable amount of time in order to diff any changes you make in your declarati…

Yeah but then you don’t have to worry about the state getting out of sync which is one of the biggest problems. Also versioning is a pain

You do because the metadata on the resource might get changed.

Re: Terraform should have remained stateless

#276
post #50

Having a state file isn't a bad idea. State files are a logical map from your code to the AWS resources. You can actually import a resource that Terraform never created into a state file so that Terraform can manage it. But of course, you could just write the code to explicitly include that pre-existing resource, rather than have to run a command to tell a state file to explicitly include it. The problem with Terrafo…

> It doesn't know how to auto-import existing resources […] It doesn't know how to detect existing resources and incorporate them into its plan. You are offerring an oversimplified view of cloud deployments. How do you propose terraform is supposed to discern between existing resources that belong to you and the ones that belong to another colleague/project? That is the first problem and is an important one as many o…

> How do you propose terraform is supposed to discern between existing resources that belong to you and the ones that belong to another colleague/project?

All of them "belong to you" if you have IAM access to the resource. It's trivial to restrict access to resources in AWS if you shouldn't be touching something. The fact that so many orgs don't use IAM properly is not a reason to make a tool unnecessarily difficult to use.

So the question isn't if it "belongs to you", the question is whether you as the user want to do something with that particular resource that you already have permissions to modify. This could be accomplished at least a half dozen ways:

  1. Is there a record in the state file of Terraform creating it? No? Then it's somebody else's resource
  2. Ask the user
  3. Command-line options
  4. The code / DSL
  5. Tags
  6. Naming convention
But it does none of these things. It just craps out with a generic error and you're left to manually fuck around with the state halfway through it already started applying changes in production.

> 1. Declarative approach to the resource management

So it's declarative. Who cares? In the real world, there are resources that Terraform didn't create that you still have to deal with. There are also resources that Terraform did create, but you lost the state file, or you moved some modules, or changed the Terraform version by mistake and can't reverse your state file version, or a million other things. "Declarative" is not some magical word that means "the real world no longer applies".

> 2. Dependency graph management – dependency management is hard.

It's not that hard. If you already have a DAG and algorithms that can manipulate objects in it, there's only like 2 or 3 more functions you need to manipulate dependencies in a graph. This is really basic CS stuff. Dependency tracking is the same thing.

> What do you mean by overwriting existing resources?

I mean if there's an AWS record that Terraform wants to create and you didn't import it, Terraform fails. This is stupid, because 1) it's not even checking if the resource already exists (it has the data providers to do this), and 2) it's making me do something manually that it could do by itself with a flag or a config or DSL entry or a prompt.

The UX is fucking ridiculous. It's like whoever wrote Terraform has never actually used it, or just doesn't give a shit about blowing up production, or getting anything done in a reasonable amount of time. Classic "humans exist to serve machines", "opinionated" tech hipster bullshit. Must be the same person who did Git's UX. And impossibly, people defend it, as if technology is supposed to make our lives harder.

Re: Terraform should have remained stateless

#277

Earlier quoted context omitted.

I built pretty much the system you’re describing for my company (a stateless terraform alternative) and this scan that you cite as a negative happens in… 100ms in parallel? Roughly the same amount of time it would take to download a state file? Dunno about you but ensuring state is always accurate and in-sync is well worth the trade-off to me.

I'm curious, and not antagonistically, about the size and variability of the infrastructure you're working with, on how many platforms it's running, and how many people are in charge of it. Is this tool private or available for us to try out?

It’s small, under 50 servers running OpenBSD and Linux under a wide variety of configurations (including gpu), 6 databases, and Redis. Though the system is easily extensible to other clouds and resource types, it only needs to work on GCP right now. Been running it in prod for the past 3 or 4 years—no outages, no downtime, no surprises. Doubled infra over the holidays then scaled everything back with no issue.

The stateless approach has worked really well for us.

Re: Terraform should have remained stateless

#278

Earlier quoted context omitted.

yeah, but then you DO have to worry about scanning every single resource you control, every time you plan and run

That does not seem problematic with modern APIs and concurrency. Two sources of truth is highly problematic

But it is problematic when you deal with larger infrastructures due to rate limits.

Re: Terraform should have remained stateless

#279

Earlier quoted context omitted.

If some state information is stored using metadata then we need that metadata to know the total state. The contents of TF state is more than what can be read from the provider APIs often.

Yeah but the extra stuff they add is a trade off and lots of people would rather not have two sources of truth to have it

Most of the people I know who use Terraform use Terraform as the source of truth.

Re: Terraform should have remained stateless

#280
Interesting read. Seems to be focusing on the neck of the woods the author is exposed to.

Terraform isn’t about cloud apis only. Terraform allows me managing keycloack realms, ssh keys, cloud resources, postgres databases, git repos, imap accounts, and so on.

The state is there to be treated as the source of truth. It gives an answer to „do I have what I want to have”. With that state, it is possible to cross reference various resource types without having to load the curent state on every run. I’m surprised that the author did not see that as a performance issue.

Imagine that you have to load all route53 state, all buckets, ec2 instances, iam roles, … on every execution, and imagine you have 200+ machines… That’s what we used to do with puppet and chef, no?

Turned out that always fetching the view of the world from an api is pretty expensive and quickly exhausts api rate limits.

It’s a pretty weak article without any effort to suggest how could it work without a state.

Post reply on HN