Earlier quoted context omitted.
> So you can in theory pick any version of your entire env, throw it at blank slate, and reliably get the environment specified by that git hash. The trap here is this only works for stateless infrastructure. If you do it with stateful resources, you'll lose all data. Your gitops tool will happily recreate EC2 instances, S3 buckets and RDS instances, all empty/initialized to whatever you defined.
There are 2 different thoughts here I think. If using GitOps in Kubernetes, then application and set up (Pods) aren’t associated with Nodes (EC2). And both can be torn down and rebuilt without state issues. When state is required, then PVCs and Stateful Sets come into play. For managed services like S3 and RDS, there are other GitOps tools like Crossplane.io which you can use for similar GitOps management. But the pa…
And that's the problematic part. With GitOps, you manage the infrastructure, so in the case of PVCs these are PVC manifests - you need to manage data separately. But even if you exclude the data, even some PVC manifest changes like size change can be tricky. Also some properties are immutable (like its storage class, access modes etc.) so that you cannot modify a PVC without recreating it.
You can decide you want to avoid the problem completely and store your data outside of Kubernetes. Now you have two problems...