It's interesting that atomic deploys weren't in from the start. That was one of the few deployment practices we really insisted on from day one at my own businesses, if only because the uncertainty you get from trying to trace problems where your system isn't in any known state makes it all but impossible to work systematically.
A related challenge where we've never really found a good solution is how to handle deploying updates atomically when both code and data model are changing. That is, we need to migrate both our application software and our database schema in some co-ordinated way.
In practice, this usually ends up being done in multiple stages, where during some intermediate part of the process we are actively maintaining both the old and new database structure and running both versions of relevant code, at some point in the process there will be a bulk conversion of existing DB data that was only in the old format to the new one, and then hopefully at the end we switch to reading only the new version, retire the old code, and if necessary remove the old DB contents that are no longer in use. Even then we probably still want to keep an implementation of our previous data API available that is reverse engineering data from the new format, just in case we have to wind back the application code due to some other problem.
I got tired just writing that, and it feels similarly dirty actually deploying it. How is everyone else handling this? Has anyone found a satisfactory way to migrate code and data forwards, and if necessary backwards, without timing or data loss issues? Controlled deployments of application code seem to be largely a solved problem with modern tools and a bit of common sense, but the database side of things doesn't seem to be nearly as clean, at least not with any of the strategies I've encountered so far.
[Edit: I see that while I was writing this, someone else has already raised a similar point elsewhere in the discussion and a few people have replied, but unfortunately only along the lines I mentioned here as well. This does not make me optimistic about finding a cleaner strategy, but further comments are still welcome.]