At an old company, we wrote this, er "model driven orchestration framework for continuous deployment":
https://github.com/tim-group/orc
Basically, there's a Git repo with files in that specify the desired versions and states of your apps in each environment (the "configuration management database").
The tool has a loops which converges an environment on what is written in the file. It thinks of an app instance as being on a particular version (old or new), started or stopped (up or down), and in or out of the load balancer pool, and knows which transitions are allowed, eg:
(old, up, in) -> (old, up, out) - ok
(old, up, out) -> (old, up, in) - no! don't put the old version in the pool!
(old, up, out) -> (old, down, out) - ok
(old, up, in) -> (old, down, in) - no! don't kill an app that's in the pool!
(old, down, out) -> (new, down, out) - ok
(old, up, out) -> (new, up, out) - no! don't upgrade an app while it's running!
Based on those rules, it plans a series of transitions from the current state to the desired state. You can model state space as a cube, where the three axes of space correspond to the three aspects of the state, vertices are states, and edges are transitions, some allowed, some not. Planning the transitions is then route-finding across the cube. When i realised this, i made a little origami cube to illustrate it, and started waving it at everyone. My colleagues thought i'd gone mad.
You need one non-cubic rule: there must be at least one instance in the load balancer at any time. In practice, you can just run the loop against each instance serially, so that you only ever bring down one at a time.
This process is safe, because if the tool dies, it can just start the loop again, look at the current state, and plan again. It's also safe to run at any time - if the environment is in the desired state, it's a no-op, and if it isn't, it gets repaired.
To upgrade an environment, you just change what's in the file, and run the loop.