[dead]
Stategraph: Terraform state as a distributed systems problem
51–60 of 68 posts
Re: Stategraph: Terraform state as a distributed systems problem
#52> The Terraform ecosystem has spent a decade working around a fundamental architectural mismatch: we're using filesystem semantics to solve a distributed systems problem. The result is predictable and painful. You portray this as a design flaw, but it's just the Hashicorp marketing funnel towards hosted Terraform, which solves the arbitration problems that you encounter at scale while allowing Hashicorp to give the c…
Terraform Cloud does not actually solve this. It moves the state into their service and manages locking centrally, but the underlying model is unchanged. It is still one file with a global lock. That is why refresh bottlenecks and contention show up there too. What we are pointing at is the model itself, not just where the file is hosted. We know this space well from building Terrateam, which competes with Terraform Cloud on orchestration.
Re: Stategraph: Terraform state as a distributed systems problem
#53Hmm I don’t see the state file as a complete downside. It is very simple and very easy to understand. It makes it easy to tell or predict what terraform will do given the current state and desired state. Its simpleness makes troubleshooting easier: the state files are easy to read and manipulate or repair in the event of a drift, mismatch, or botched provider update. With the solution proposed it feels like the state…
> easy to tell or predict what terraform will do
predict is the operative word there, because Terraform is so disconnected from the underlying provider's mental model that it is the expression "no plan survives first contact with the enemy" made manifest
Now, I am one million percent open to the pushback that "well, that's a provider's problem" but I also can't easily tell if they are operating within the bounds of TF's mental model, or is it literally that every provider ever is just that lazy?
Re: Stategraph: Terraform state as a distributed systems problem
#54Very interesting. I found myself nodding YES the whole way through the post. Something like this could lead to a large shift in how we manage infrastructure. We split terraform configs for more reasons than just splitting state of course, but something like this could make other approaches to organizing things more viable. Really cool and will be keeping an eye on this.
Re: Stategraph: Terraform state as a distributed systems problem
#55This looks awesome but what is the end state (pun intended) going to be? Will this be an open source project? Commercial closed-source? Somewhere in between?
Re: Stategraph: Terraform state as a distributed systems problem
#56This is the least of the problems. The big problem is the shared cache. Supposedly OpenTofu fixed it but Terraform proper is still not allowing multiple Terraform plans and/or applies in parallel over the same cache. Atlantis claimed that it fixed it, but it didn't. So pathetic!
Re: Stategraph: Terraform state as a distributed systems problem
#57This is very cool. I love the idea of querying the state, and it opens up a ton of very easy reporting options.
Disagree since you are relying on non authoritative state to be correct. This can be a path to madness.
Re: Stategraph: Terraform state as a distributed systems problem
#58Hey! One of the Stategraph developers here and can answer any questions. The major motivation is just how small scale Terraform/Tofu start to breakdown and creates work for users when they have to refactor for performance issues that shouldn't exist. So we want a drop in solution that just dissolves those issues without the user having to do anything.
[deleted]
Re: Stategraph: Terraform state as a distributed systems problem
#59I can't help but wonder whether the problem being addressed is the result of two antipatterns.
The first is that the scope being managed by a single Terraform application is too broad (e.g., thousands of resources instead of tens or hundreds). File-level locking is fine for small databases with few to no concurrent writers, but as more users come in, and the database gets bigger, you need record-level locking. For Terraform state files, it begs the question why the database got so big and why there were so many concurrent users in the first place.
Second, Terraform state files are a cache but they're being mistreated as a source of truth. This isn't the user's fault but it is the result of (understandable) impatience which results in inevitable shortcut-seeking. It's been a risk since Terraform's inception, and it won't go away as long as people complain that collecting current actual state from the resource provider is too slow.
Re: Stategraph: Terraform state as a distributed systems problem
#60This is cool and useful for teams with huge states and slow locks. Terraform, while great, still has lots of room for improvement.
I keep wondering though if split state files isn’t a good thing in some sense anyway. It isolates a lot of problems. Access, obviously. Another common theme is that applying tends to be a game of roulette, even if you haven’t changed anything. Cloud vendor added or renamed something in their backed. Provider update made some field deprecated. Expected state drift that wasn’t properly marked as ”ignore_changes”. Unexpected state drift by your over-excited intern. When I as an app-developer apply a simple config file change, I really don’t want to be bothered about dirty state in the networking backbone that I understand nothing about.
It’s also not entirely clear how the solution keeps track of the desired state if multiple actors are changing it at the same time. Wouldn’t one successful apply make the next person, who don’t have your local changes, revert it back on first apply? Does this expect heavy use of resource targeting?