Once you have pushed code to production and realized there is an issue, how do you roll back?
Do you roll forward? Flip DNS back to the old deployment? Click the button in heroku that takes you back to the previous version?
Ask HN: How do you roll back production?
1–10 of 156 posts
Re: Ask HN: How do you roll back production?
#2Specifically, I tell Jenkins to deploy the commit hash that was last known good. Jenkins just deploys, and doesn't really know that it's a "roll back."
Generally, going back to a known clean state should be easier, safer and relatively quick (DNS flip is fast, redeploy of old code is fast if your automation works well).
In some cases changes to your data may make rolling back cause even more problems. I've seen that happen and we were stuck doing a rapid hot fix for a bug, which was ugly. We did a lot more review to ensure we avoided breaking roll back. So I'd advise code review and developer education of that risk.
Re: Ask HN: How do you roll back production?
#3Specifically, I tell Jenkins to deploy the commit hash that was last known good. Jenkins just deploys, and doesn't really know that it's a "roll back." Generally, going back to a known clean state should be easier, safer and relatively quick (DNS flip is fast, redeploy of old code is fast if your automation works well). In some cases changes to your data may make rolling back cause even more problems. I've seen that…
How do you find the commit hash that is the last known good? Looking through jenkins release logs, asking someone, something else?
Re: Ask HN: How do you roll back production?
#4Specifically, I tell Jenkins to deploy the commit hash that was last known good. Jenkins just deploys, and doesn't really know that it's a "roll back." Generally, going back to a known clean state should be easier, safer and relatively quick (DNS flip is fast, redeploy of old code is fast if your automation works well). In some cases changes to your data may make rolling back cause even more problems. I've seen that…
How do you find the commit hash that is the last known good? Looking through jenkins release logs, asking someone, something else?
having the jenkins build history reflect release deployments generally works well enough.
if build #20 is bad, build #19 deployed with hash XYZ was last known good, click 'rebuild' or 'replay' and it'll deploy #19 again.
Re: Ask HN: How do you roll back production?
#5Specifically, I tell Jenkins to deploy the commit hash that was last known good. Jenkins just deploys, and doesn't really know that it's a "roll back." Generally, going back to a known clean state should be easier, safer and relatively quick (DNS flip is fast, redeploy of old code is fast if your automation works well). In some cases changes to your data may make rolling back cause even more problems. I've seen that…
How do you find the commit hash that is the last known good? Looking through jenkins release logs, asking someone, something else?
In our case its quite simple. All commits should be tested and if we need to roll back, it means either that tests have failed and it was pushed nonetheless, tests were missing or tests didn't catch the issue.
In the first two cases, we revert to the commit where untested or failed code was introduced to the master. This basically never happens. In the third case, you need to do some debugging and try to figure out why it's broken and either fix it or revert it. Basically just look at the git history. If the code has dependencies then you might need to do code triage and produce a hotfix. If it's a relatively isolated piece, then just revert and fix it at a better time.
We use semver and gitlab's tags so we know just by the versioning if the code that is broken is important or not and if we can roll back.
Re: Ask HN: How do you roll back production?
#6Tell people we need to roll back, clone the repo to my hard drive, open up git, undo the commit that merged the bad code in, push it. All done.
"Production"? Does that mean something that goes to the customers? Very few of our customers keep up with releases so it's generally not a big deal. We can have a release version sitting around for weeks before any customer actually installs it; some customers are happy with a five year old version and occasional custom patches.
I bet it's a bigger problem for those for whom the product is effectively a running website, but those of us operating a different software deployment model have a different set of problems.
Re: Ask HN: How do you roll back production?
#7At my workspace, we use Gitlab. We pick up the old(stable) Job ID and ask Devops to deploy it.
Re: Ask HN: How do you roll back production?
#8Earlier quoted context omitted.
How do you find the commit hash that is the last known good? Looking through jenkins release logs, asking someone, something else?
In our case its quite simple. All commits should be tested and if we need to roll back, it means either that tests have failed and it was pushed nonetheless, tests were missing or tests didn't catch the issue. In the first two cases, we revert to the commit where untested or failed code was introduced to the master. This basically never happens. In the third case, you need to do some debugging and try to figure out w…
What about logical errors?
math.pow(2, 4) vs math.pow(4, 2)
Re: Ask HN: How do you roll back production?
#9Rolling back is usually a bad practice and can get quite challenging if not impossible in distributed environments.
If you can pinpoint a specific commit that is causing the issue. Revert that commit and go through your standard release process.
Re: Ask HN: How do you roll back production?
#10roll back (step back), is an inherited from waterfall anti-pattern.
Now we should only march forward with small, on demand releases, this way we will know exactly where the issue is and will be able to fix it forward quickly.
Rollbacks were a strategy with monthly (or even quarterly [insane huh?]), giant, stinky, release dumps, knowing there is no way we could quickly identify and deploy the fix. aka lets throw production 3 months back and take another 2 month for figuring out there the issue that happened during last release is.
And to finally answer your question: we never roll back. We always march forward.