Having done some work with building graph representation of compute resources (by importing tfstate & syncing from aws) and its a very useful representation for building things like visualizations and detecting dependencies. Moving tfstate to a real distributed graph system would help solve a LOT of the nasty hacks they mention in this article that I have run into HARD when working with very large numbers of resources and multiple teams/team sizes.
Stategraph: Terraform state as a distributed systems problem
41–50 of 68 posts
Re: Stategraph: Terraform state as a distributed systems problem
#42I don't think a DAG makes sense as a model here. I've always likened terraform apply to setting what would be called a "waypoint" in robotics, or a "keyframe" in animation. The system doesn't immediately change state to the new state all at once, but it starts to tend towards the new desired state.
While the intermediary states might form a graph when considering different subsystems, the desire to move towards a given state should probably appear serial over time. When you make a terraform transaction, you have to coordinate in a serializable way with everyone else managing the desired state, even though the actual state will be messy along the way.
Re: Stategraph: Terraform state as a distributed systems problem
#43I don't think a DAG makes sense as a model here. I've always likened terraform apply to setting what would be called a "waypoint" in robotics, or a "keyframe" in animation. The system doesn't immediately change state to the new state all at once, but it starts to tend towards the new desired state. While the intermediary states might form a graph when considering different subsystems, the desire to move towards a giv…
Your Terraform/Tofu code is a graph. Resources are nodes and the dependencies between them are edges, so a graph is a very natural and useful representation of infrastructure. There are well-understood algorithms for transitioning a graph between two representations that can be leveraged as well. I think how you describe representing state would work fine, but a graph works quite well and is more natural for the types of operations we want to perform on the graph.
Re: Stategraph: Terraform state as a distributed systems problem
#44I deeply believe that the whole "state" approach with TF is flawed. You end up doing a 3-way merge between the actual state, the desired state, and the recorded state every time you try to make changes.
Long time ago, I was simply doing stuff like this:
> resources = describe_resources_by_tag(env_name=env, some_tag=tag) > if resource_doesnt_exist(resources, some_resource): > create_resource(resource)
This was very robust and easy to explain. You look around the system, using some tag-based filtering (in AWS, GCP, Azure) and then perform actions to bring the system to the desired state.
Re: Stategraph: Terraform state as a distributed systems problem
#45[deleted]
Re: Stategraph: Terraform state as a distributed systems problem
#46Very cool. Biggest question I have is how users with large setups where Terraform state has already been split would migrate to this. Would existing state blobs be namespaced into the One True State Graph in PG? Would Stategraph know how to merge different state blobs that are pointing to the same real-life resources? Will Stategraph promote some kind of convention for how state should be named, so that new projects…
The high-level vision of Stategraph is that the entire world's infrastructure should be representable as a single root module, with proper isolate and RBAC. It should scale and be secure. With that, the way Stategraph works best is everything being in a single repo and in a single root module.
Additionally, Stategraph should Just Work with your existing TF codebase. You important the state and you're off to the races.
Once all of your state is in Stategraph, though, moving state around really becomes a question of what name those piece of state should have. So, if you want to merge two root modules, it could be the case that you can check "do my resource names overlap?" If no, you can tell Stategraph to merge the states and then copy your code into a single root module and go. Otherwise, you need to do some resource renaming.
While we don't have all the details in place, I think it is quite likely that Stategraph will support metadata on your resources, perhaps with a new block. This way you could provide namespaces to collections of resources, and that could make merging even easier. But, there is a bit to figure out before that is a reality or determined to be the best way to go.
Re: Stategraph: Terraform state as a distributed systems problem
#47> The Terraform ecosystem has spent a decade working around a fundamental architectural mismatch: we're using filesystem semantics to solve a distributed systems problem. The result is predictable and painful.
You portray this as a design flaw, but it's just the Hashicorp marketing funnel towards hosted Terraform, which solves the arbitration problems that you encounter at scale while allowing Hashicorp to give the cli tooling away for free.
Re: Stategraph: Terraform state as a distributed systems problem
#48As someone who worked on projects which "solved" this issue by basically having one stack in one repo by resource sub-set (an ALB fronting an ASG will result in one repo for the ALB and one repo for the ASG), I can only welcome any solution trying to simplify team collaboration on Terraform stacks!
Re: Stategraph: Terraform state as a distributed systems problem
#49Are there any statistics/analyses for the popularity of these different configuration management languages/frameworks (Terraform, Pullumi etc) in cloud settings? Trying to figure out which one(s) are worth learning.
I believe the DORA report has some information on this. Terraform/Tofu dominate, by far.
Re: Stategraph: Terraform state as a distributed systems problem
#50If you're at the point of managing thousands of resources inside a single statefile, and that makes the most sense to your setup: you've outgrown terraform.
From my experience the problem at scale isn't that Terraform stops being useful. The problem is how state gets managed. Every IaC approach still has to coordinate changes across shared resources and none of them escape that. Other tools just shift the trade offs around. In house systems usually end up rebuilding the same thing in another form. At scale the choices are pretty simple. You split state and live with orch…
Changes across shared resources should be few and far between. Terraform really should only be used to setup long-lived resources, such as your VPC, initial IAM, and a bootstrapping system/management plane (eg, your kubernetes cluster). Once your infrastructure is up and running, further operations should be api-driven (aka controllers).
I'm not really a fan of crossplane, it's much simpler to roll your own custom operator, especially now that things like the Azure Service Operator exist (I think there's something equivalent for aws as well). This gives you a lot more flexibility for writing unit tests for your business logic.