The more layers of automation we add, the more invisible points of failure. Magic is great until it isn’t. I feel their pain. The sinking feeling when you realize that data is gone and not coming back is an awful experience.
The more layers of automation we add, the more invisible points of failure.
Without automation this situation would've played out like: well we got used to running things with e.g. --force or --yes or just hitting yes manually at every prompt. Unfortunately we just nuked our data store.
Alternatively they would've looked at the dashboards, seen perhaps low CPU or memory utilization for the data store namespace and manually nuked it to save some money..
While this smells like a process issue it's mostly an architectural one. It's good that things were namespaced, however, for persistent data the infra should be more or less air gapped from a tooling POV. Updating the persistent data store should be a whole separate CI job, ideally with more intervention required to effect change.
