Ask HN: How do you roll back production?

Cost can be a killer here though. If you're flipping from blue to green and vice versa you either have to have capacity in stand-by(expensive) or spin up new capacity before flipping(time-consuming).

My almost universal experience has been to simply do a Git revert and let the CI pipeline do its thing. Pros - It's simple. Cons - It's slow, especially in an emergency.

My last job had an extra layer of security. As a .net house all new deployments were sent to Azure in a zip file. We backed those up and maintained FTP access to the Azure app service. If a deployment went really wrong and we couldn't wait the 10-20 mins for the CI pipline to process a revert, we'd just switch off the CI process and FTP upload the contents of the previous last good version.

Of course, if there were database migrations to deal with then all hell could break loose. Reverting a DB migration in production is easier said than done especially if a new table or column has already started being filled with live data.

To be fair though, most of the problems I encountered were usually as the result of penny pinching by management who didn't want to invest in proper deployment infrastructure.

I'm strictly against rollbacks and I'm strictly for everything continuous.

If I need to do a roll back it means that I don't trust the system nor the code base. I will do the roll back but after that there will be a very productive retro about how we can do better to avoid rollbacks in the future (aka what did we learn).

But again, as I said, there is a place and time for everything! And there are many variables! Even how you structure your teams affects deployments, engineering culture, engineering team types (cross-functional, generalized; specialized etc), if the team that makes a decision about the roll back is not the team that introduced the bug.

My approach is not dogmatic (have your standardized roll backs if those work best for your company, release cycles, teams) it's idealistic (that's what I aim for, personally)

I suspect we're going to agree to disagree here, but I highly advise you to re-consider the idea of framing a roll back as an unforced failure to your team. The last dynamic you want in a retrospective is one where not only did an unexpected failure happen (a bug pushed to production), but then the team collectively 'let you down' by pulling the rollback lever, instead of thinking and working harder on fixing the issue live. In such a scenario you're forcing people to feel they need to "cry uncle" when they can't solve the problem quickly, and putting themselves into the middle of a conflict of interest between making a well-tested, reviewed change that is sure to fix the problem, and rolling the dice on a quick fix in the hope it'll reduce the total outage. That's not the recipe for a positive, blameless culture.

When we roll back on my team, it's uncommon but when it happens it's considered a success if it was made through a systematic decision-making process. Making a sane decision in the interest of our users to restore service quickly is always a win. I can assure you, it does not compromise your ability to do continuous delivery or small changes by having and occasionally using a rollback mechanism. If you are fearful of the idea that having such a mechanism and plan in place somehow will lead to people questioning your principles in a way you cannot defend, then that is a separate problem, since the two things you mention that are incompatible are in fact compatible and highly defensible.

It is not a legacy from "waterfall" or any of the other things you mention, because your claim can be refuted through a single counter example, and I've worked on 3 separate projects where such counter examples exist: we had a rollback method, it was used once in a while, and we shipped changes to production multiple times a day using continuous delivery. At no point on these projects did the ability or use of roll back lead to some kind of hard-to-explain loss in delivery velocity. On the contrary, I suspect if that mechanism did not exist, several failures that were easy to get back to green would have turned into a toxic hellhole, and my team mates would have been much more fearful around shipping, which is the high order bit when it comes to velocity and embracing continuous delivery of small changes.

Before implementing CI with containers I used to deploy using Capistrano. One thing I loved about this setup was that in case of needing to rollback, I would just run a command which would change a symlink pointing to the previous deploy and restart. All usually done in a couple of seconds.