Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you roll back production?

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 156 posts

Re: Ask HN: How do you roll back production?

#11

Post by drubenstein »

Depends on the issue - if there's a code / logic bug that doesn't affect the data schema, we use elastic beanstalk versions to go back the previous version (usually this is a rolling deploy backwards), and then clean up the data manually if necessary. Otherwise, we roll forward (fix the bug, clean up the data, etc).

It's more often been the case for us that issues are caused by mistaken configuration / infrastructure updates. We do a lot of IAC (Chef, Cloudformation), so with those, it's usually a straight git revert and then a normal release.

Re: Ask HN: How do you roll back production?

#12

Post by ISL »

rinchik wrote: post #10

roll back (step back), is an inherited from waterfall anti-pattern. Now we should only march forward with small, on demand releases, this way we will know exactly where the issue is and will be able to fix it forward quickly. Rollbacks were a strategy with monthly (or even quarterly [insane huh?]), giant, stinky, release dumps, knowing there is no way we could quickly identify and deploy the fix. aka lets throw produ…

Until the fix is identified, can't one 'march forward' by rolling back recent changes to a known-good state?

Re: Ask HN: How do you roll back production?

#13

Post by mooreds »

rinchik wrote: post #10

roll back (step back), is an inherited from waterfall anti-pattern. Now we should only march forward with small, on demand releases, this way we will know exactly where the issue is and will be able to fix it forward quickly. Rollbacks were a strategy with monthly (or even quarterly [insane huh?]), giant, stinky, release dumps, knowing there is no way we could quickly identify and deploy the fix. aka lets throw produ…

Congrats, that sounds awesome. So when a bug affects prod, do you:

* Find the code that is affected

* Write a test

* Have it go through ci/cd

* Deploy to prod.

Or is there a different way of deploying a big priority bugfix to production than normal deploys?

Re: Ask HN: How do you roll back production?

#14

Post by mooreds »

rvdmei wrote: post #9

Rolling back is usually a bad practice and can get quite challenging if not impossible in distributed environments. If you can pinpoint a specific commit that is causing the issue. Revert that commit and go through your standard release process.

Would love to know more about how you roll forward in that case. Or do you canary so thoroughly that bugs never get to prod?

Re: Ask HN: How do you roll back production?

#15

Post by dxhdr »

Push different code to production, either the last-known-good commit, or new code with the issue fixed.

I imagine that much larger operations likely do feature flags or a rolling release so that problems can be isolated to a small subset of production before going wide. But still the same principle, redeploy with different code.

Re: Ask HN: How do you roll back production?

#16

Post by ISL »

StreamBright wrote: post #8
httpsterio wrote: post #5

Earlier quoted context omitted.

In our case its quite simple. All commits should be tested and if we need to roll back, it means either that tests have failed and it was pushed nonetheless, tests were missing or tests didn't catch the issue. In the first two cases, we revert to the commit where untested or failed code was introduced to the master. This basically never happens. In the third case, you need to do some debugging and try to figure out w…

What about logical errors? math.pow(2, 4) vs math.pow(4, 2)

A test can test for those: "does the code give mathematically correct results?"

Re: Ask HN: How do you roll back production?

#17

Post by ccmcarey »

robbya wrote: post #2

Specifically, I tell Jenkins to deploy the commit hash that was last known good. Jenkins just deploys, and doesn't really know that it's a "roll back." Generally, going back to a known clean state should be easier, safer and relatively quick (DNS flip is fast, redeploy of old code is fast if your automation works well). In some cases changes to your data may make rolling back cause even more problems. I've seen that…

How do you deal with DB migrations?

Re: Ask HN: How do you roll back production?

#19

Post by atemerev »

Blue-green deployment is the only way to fly: https://martinfowler.com/bliki/BlueGreenDeployment.html

There are two identical prod servers/cloud configurations/datacenters: blue and green. Each new version is deployed intermittently on blue and green areas: if version N is on blue, version N-1 is on green, and vice versa. If some critical issue happens, rolling back is just switching the front router/balancer to the other area, which can be done instantly.

Re: Ask HN: How do you roll back production?

#20

Post by kostarelo »

rinchik wrote: post #10

roll back (step back), is an inherited from waterfall anti-pattern. Now we should only march forward with small, on demand releases, this way we will know exactly where the issue is and will be able to fix it forward quickly. Rollbacks were a strategy with monthly (or even quarterly [insane huh?]), giant, stinky, release dumps, knowing there is no way we could quickly identify and deploy the fix. aka lets throw produ…

I totally get your point but this approach seems a bit dogmatic to me. Even with modern CI/CD techniques and agile methodologies in place, rolling back could still be the best choice.

For example, always marching forward means that any time an issue is coming up, certain resources must be allocated in tackling the issue. That can't always be the case.

Smaller and more frequent releases are preferable and most of the time a single line change will fix the issue, but other times rolling back may be the best option.

Post reply on HN