Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you roll back production?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 156 posts

Re: Ask HN: How do you roll back production?

#21

Post by sqldba »

EliRivers wrote: post #6

Tell people we need to roll back, clone the repo to my hard drive, open up git, undo the commit that merged the bad code in, push it. All done. "Production"? Does that mean something that goes to the customers? Very few of our customers keep up with releases so it's generally not a big deal. We can have a release version sitting around for weeks before any customer actually installs it; some customers are happy with…

You’re being downvoted but it’s a totally reasonable reply for normal vendor software.

Re: Ask HN: How do you roll back production?

#23

Post by emptysea »

Ideally we'd have the problematic code behind a feature flag and we'd turn the flag off.

For other issues we press the rollback button in the Heroku dashboard.

Heroku has its problems: buildpacks, reliability, cost, etc, but the dashboard deploy setup is pretty nice.

Re: Ask HN: How do you roll back production?

#24

Post by cddotdotslash »

We actually just implemented something like this. Our entire environment is AWS CodeBuild, CodePipeline, and Lambda-based, but the process would be similar for more traditional environments:

1. Developer creates a PR. To be mergeable, it must pass code review, be based on master, and be up-to-date with master (GitHub recently made this really easy by adding a one-click button to resync master into the PR).

2. Each commit runs a build system that installs dependencies, runs tests, and ZIPs the final code to an S3 bucket.

3. Once the developer is ready to deploy, and the PR passes the above checks, they type "/deploy" as a GitHub comment.

3. A Lambda function performs validation and then updates our dev Lambda functions with the ZIP file from S3. Once complete, it leaves a comment on the PR with a link to the dev site to review.

4. The developer can now comment "/approve" or "/reject". Reject reverts the last Lambda deploy in dev. Approve moves the code to stage.

5. The above steps repeat for stage --> prod.

6. Once the code is in prod, the developer must approve or reject. If rejected, the Lambdas are reverted all the way back through dev. If approved, the PR is merged by the bot (we have some additional automation here, such as monitoring CloudWatch metrics for API stability, end-to-end tests, etc).

TL;DR - Don't merge PRs until the code is in production and reviewed. If a rollback is needed afterwards, create a rollback (roll-forward) PR and repeat.

Re: Ask HN: How do you roll back production?

#25

Post by gfodor »

atemerev wrote: post #19

Blue-green deployment is the only way to fly: https://martinfowler.com/bliki/BlueGreenDeployment.html There are two identical prod servers/cloud configurations/datacenters: blue and green. Each new version is deployed intermittently on blue and green areas: if version N is on blue, version N-1 is on green, and vice versa. If some critical issue happens, rolling back is just switching the front router/balancer to the…

Yup, can't beat blue/green. A big part of the reason I recommend it is because unlike most other rollback schemes, the actual mechanism for rolling back is the same as for a normal release (a load balancer flip), so it's continually exercised and validated. To roll back, you literally do a deploy, but you just skip the step where you alter the bits on the dark cluster. This is highly unlikely to fail, since the only thing that has to not go wrong is the part where the software update is skipped, which basically boils down to a conditional.

Any mechanism for rollbacks that isn't tested continuously is likely to fail during incident response. It's a huge anti-pattern to have 'dark' processes only used during incident response -- same thinking behind why you should also be continually testing your backups, continuously killing servers to verify recovery, etc.

Re: Ask HN: How do you roll back production?

#26

Post by rinchik »

mooreds wrote: post #13
rinchik wrote: post #10

roll back (step back), is an inherited from waterfall anti-pattern. Now we should only march forward with small, on demand releases, this way we will know exactly where the issue is and will be able to fix it forward quickly. Rollbacks were a strategy with monthly (or even quarterly [insane huh?]), giant, stinky, release dumps, knowing there is no way we could quickly identify and deploy the fix. aka lets throw produ…

Congrats, that sounds awesome. So when a bug affects prod, do you: * Find the code that is affected * Write a test * Have it go through ci/cd * Deploy to prod. Or is there a different way of deploying a big priority bugfix to production than normal deploys?

Las 2 steps are merged (ci/cd is part of the deployment to prod, but generally yes.

Also worth to note, priority bug fix is not really about pipelines it's more about the ability to dynamically reallocate resources. Depending on the complexity of the affected area we should be able allocate as many devs as it is useful to fixing it. (similar to "Fast Lane" in Kanban)

Re: Ask HN: How do you roll back production?

#27

Post by gfodor »

rinchik wrote: post #10

roll back (step back), is an inherited from waterfall anti-pattern. Now we should only march forward with small, on demand releases, this way we will know exactly where the issue is and will be able to fix it forward quickly. Rollbacks were a strategy with monthly (or even quarterly [insane huh?]), giant, stinky, release dumps, knowing there is no way we could quickly identify and deploy the fix. aka lets throw produ…

Shorter releases can help with reducing the difficulty of immediately addressing problems, but it's an error to equate the reduction of risk as an elimination of risk.

There are always going to be failure modes that require extensive time to diagnose and debug, even with small changes being made. Additionally, you want that diagnostic phase to happen without time pressure. If you do not have a sane rollback mechanism to use in those scenarios, you are doing a disservice to your users and your team.

Your users suffer, because the outage or breakage will last as long as it takes for you to address the underlying issue directly, instead of just rolling back to restore service. They will be forced to hear frustrating things like "we're working on it", since you don't know what's wrong yet, when instead you could have just rolled back before most users even noticed there was a problem.

And, more importantly, your team will suffer greatly, because they will be forced to work under pressure when an incident like this arises. And, worse, they will also 'learn' that accidentally pushing breaking changes to production results in an extremely unpleasant and toxic situation for everyone, leading to systemic fear-of-deploys and undermining a blameless culture.

So you should have a rollback mechanism that is solid, tested, and easy to use for scenarios where a non-trivial regression or outage arises in production, even if you are doing continuous delivery of small patches.

Re: Ask HN: How do you roll back production?

#28

Post by rinchik »

ISL wrote: post #12
rinchik wrote: post #10

roll back (step back), is an inherited from waterfall anti-pattern. Now we should only march forward with small, on demand releases, this way we will know exactly where the issue is and will be able to fix it forward quickly. Rollbacks were a strategy with monthly (or even quarterly [insane huh?]), giant, stinky, release dumps, knowing there is no way we could quickly identify and deploy the fix. aka lets throw produ…

Until the fix is identified, can't one 'march forward' by rolling back recent changes to a known-good state?

Nope, you are "pretending" that a step back is a step forward (which is not true). I never had to roll back anything within last couple years and very happy about that.

Also note, that roll back was a valid strategy back in the day and still can be useful tool in your garage of tools. It can be useful when, for example, dealing with complex legacy systems that were created decades ago, or complex systems developed by outsourced development teams. You'll know when roll back is useful when you see it.

Re: Ask HN: How do you roll back production?

#29

Post by wickedOne »

interesting question.

we roll forward and thus far never ran into the situation that that wasn't possible in a reasonable amount of time.

nevertheless i've wondered more than once what would happen if we run into such a situation and there's a substantial database migration in the process (i.e. with table drops).

curious to learn what the different strategies are on that point: do you put your table contents in the down migration, do you revert to the last backup, etc.

Re: Ask HN: How do you roll back production?

#30

Post by perlgeek »

We use https://gocd.io/ for our build + deployment pipelines. A rollback is just re-running the deployment stage of the last known-good version.

Since the question of database migrations came up: We take care to break up backwards incompatible changes into multiple smaller ones.

For example, instead of introducing a new NOT NULL column, we first introduce it a NULLable, wait until we are confident that we don't want to roll back to a software version that leaves the column empty, and only then changing it to NOT NULL.

It requires more manual tracking than I would like, but so far, it seems to work quite well.

Post reply on HN