Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you roll back production?

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 156 posts

Re: Ask HN: How do you roll back production?

#31

Post by Angostura »

gfodor wrote: post #25
atemerev wrote: post #19

Blue-green deployment is the only way to fly: https://martinfowler.com/bliki/BlueGreenDeployment.html There are two identical prod servers/cloud configurations/datacenters: blue and green. Each new version is deployed intermittently on blue and green areas: if version N is on blue, version N-1 is on green, and vice versa. If some critical issue happens, rolling back is just switching the front router/balancer to the…

Yup, can't beat blue/green. A big part of the reason I recommend it is because unlike most other rollback schemes, the actual mechanism for rolling back is the same as for a normal release (a load balancer flip), so it's continually exercised and validated. To roll back, you literally do a deploy, but you just skip the step where you alter the bits on the dark cluster. This is highly unlikely to fail, since the only…

They you just have to worry about rolling back the changed database schema that you have rolled out to both green and bluie variants - I presume. That bit feels tricky

Re: Ask HN: How do you roll back production?

#32

Post by rvdmei »

mooreds wrote: post #14
rvdmei wrote: post #9

Rolling back is usually a bad practice and can get quite challenging if not impossible in distributed environments. If you can pinpoint a specific commit that is causing the issue. Revert that commit and go through your standard release process.

Would love to know more about how you roll forward in that case. Or do you canary so thoroughly that bugs never get to prod?

Bugs do get to prod, that’s just reality. If you keep changes small and release often you won’t see many big bugs in production. We typically keep master in an “always deployable” state. If a bug gets found in prod, we either fix it or create a new commit that reverts the commit that caused the problem. If it’s a faulty migration, reverting the commit usually won’t work and we have to fix the bug. For QA we use a mix of automated tests and manual tests.

Re: Ask HN: How do you roll back production?

#33

Post by gfodor »

Angostura wrote: post #31
gfodor wrote: post #25

Earlier quoted context omitted.

Yup, can't beat blue/green. A big part of the reason I recommend it is because unlike most other rollback schemes, the actual mechanism for rolling back is the same as for a normal release (a load balancer flip), so it's continually exercised and validated. To roll back, you literally do a deploy, but you just skip the step where you alter the bits on the dark cluster. This is highly unlikely to fail, since the only…

They you just have to worry about rolling back the changed database schema that you have rolled out to both green and bluie variants - I presume. That bit feels tricky

Yep, that's a fair point. In scenarios where you have made non-backwards compatible database changes, this approach cannot be used. Typically the dark cluster is accessible for pre-flight testing, so you can run tests there to determine if the service is still operating as expected despite the database being one migration step ahead. In your response runbook, you should have a contingency for the scenarios where that happens. (For many projects, the % of changes that have this characteristic are small enough that it doesn't move the needle on the risk profile, but Murphy's law can mess with you :))

Re: Ask HN: How do you roll back production?

#34

Post by rinchik »

gfodor wrote: post #27
rinchik wrote: post #10

roll back (step back), is an inherited from waterfall anti-pattern. Now we should only march forward with small, on demand releases, this way we will know exactly where the issue is and will be able to fix it forward quickly. Rollbacks were a strategy with monthly (or even quarterly [insane huh?]), giant, stinky, release dumps, knowing there is no way we could quickly identify and deploy the fix. aka lets throw produ…

Shorter releases can help with reducing the difficulty of immediately addressing problems, but it's an error to equate the reduction of risk as an elimination of risk. There are always going to be failure modes that require extensive time to diagnose and debug, even with small changes being made. Additionally, you want that diagnostic phase to happen without time pressure. If you do not have a sane rollback mechanism…

Well, highly unlikely that such an error will arise. If it does and you know that you actually need to roll back then most likely something else is wrong.

But again, I also saw the other comment about how "Dogmatic" my approach is. I wouldn't say it's dogmatic, idealistic - yes. But not dogmatic. There is a place and time for anything and roll back can STILL be useful when you don't trust the system nor the code base (as I pointed in my other comment, rollbacks are useful with legacy systems and systems that you have to maintain that were build by outsourced teams).

Well also roll backs is the first thing you think of when you join a large company as a director of engineering to support systems you never touched before.

Re: Ask HN: How do you roll back production?

#35

Post by cdumler »

You are asking a very generic question without really stating what environment you are using. The ability to "rollback" is really a statement of how you have defined your deployments. A good environment really should have a few things:

* A version control system (ie. git) that has a methodology for controlling what is tested and then released (ie. feature releases). If you want the ability to revert a feature, you need to use your version control to group (ie. squish) code into features they can can be easily reverted. Look up the GIT Branching Model [1]. It's a good place to start when thinking about organizing your versioning to control releases.

* You should be able to deploy from any point in your version control. Make sure your deployment system is able to deploy from a hash, tag or branch. This gives you the option of "reverting" by deploying from a previously known good position. I would highly suggest automating deployment to generate timestamp tags into the repo for deployment so you can see the history of deployments.

* Try to make your deployments idempotent and/or separate your state changes so they can be independently controlled. If you have migrations, make sure they can withstand being deployed again, ie. "DROP TABLE IF EXISTS" then "CREATE TABLE", so redeploying doesn't blow up. If you need to roll back, you can rollback as much as you need to the point you want to deploy. A trait of a well designed system is it needs few state changes to add new features and/or those state changes can be easily controlled.

* Have a staging system(s). You should be able to deploy to a staging system to verify the behavior of a deployment. It should be able replicate the production every way except in data content. Ideally, should also build this from scratch every time so that you can guarantee if production dies hard death you can completely reproduce it. A great system will also do this for production, bring it up for final testing, and then you can switch over to it once tested.

Notice the trend here is to breakup the dependences between how, what, and where code is deployed so that have many ways to respond to issues. Maybe the solution is small enough to just make fix in the future. Maybe it is create an emergency patch, test it on a new production deployment and then switch over. Maybe it is so bad you want to immediately deploy a previous version and get things running again. All of these abilities depend on building your system such that you have these choices.

[1] https://nvie.com/posts/a-successful-git-branching-model/

Re: Ask HN: How do you roll back production?

#37

Post by gfodor »

rinchik wrote: post #34
gfodor wrote: post #27

Earlier quoted context omitted.

Shorter releases can help with reducing the difficulty of immediately addressing problems, but it's an error to equate the reduction of risk as an elimination of risk. There are always going to be failure modes that require extensive time to diagnose and debug, even with small changes being made. Additionally, you want that diagnostic phase to happen without time pressure. If you do not have a sane rollback mechanism…

Well, highly unlikely that such an error will arise. If it does and you know that you actually need to roll back then most likely something else is wrong. But again, I also saw the other comment about how "Dogmatic" my approach is. I wouldn't say it's dogmatic, idealistic - yes. But not dogmatic. There is a place and time for anything and roll back can STILL be useful when you don't trust the system nor the code base…

You didn't really address my points. Its hard to quantify just how "highly unlikely" a failure is, but its your job as a systems designer to build systems that are robust under a wide variety of unlikely failure scenarios. Not having rollbacks results in a system that is extremely problematic in those unlikely scenarios where a quick fix cannot be immediately addressed. Not to mention, rolling forward under such a regime has its own unique risk: since the 'fix' was made under pressure since there was no alternative to restore service quickly, it's often the case that simple human errors get introduced when rolling forward. I've seen it a million times.

In my experience, a healthy incident response process has a fork in the decision tree at the very top: do we roll back, or do we attempt to fix live? And in the latter case, we time box how long we're willing to spend, and defer to rolling back for all but the most trivial, obvious fixes. Even if you don't use rollback often, having that top level fork is a release valve for all of the toxic implications I mentioned in the scenario where you do actually need it.

Even if you have several dozen incidents happen where you didn't need it a black swan event will eventually show up -- and that event will be the one that will have the lasting impact on your company's public perception and the morale of your team.

Re: Ask HN: How do you roll back production?

#39

Post by EliRivers »

sqldba wrote: post #21
EliRivers wrote: post #6

Tell people we need to roll back, clone the repo to my hard drive, open up git, undo the commit that merged the bad code in, push it. All done. "Production"? Does that mean something that goes to the customers? Very few of our customers keep up with releases so it's generally not a big deal. We can have a release version sitting around for weeks before any customer actually installs it; some customers are happy with…

You’re being downvoted but it’s a totally reasonable reply for normal vendor software.

Something I remind myself of regularly is that the entire software world I have ever experienced is actually a tiny, tiny piece of the real software world.
Post reply on HN