Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you roll back production?

news.ycombinator.com

Post reply on HN
Print view View original

101–110 of 156 posts

Re: Ask HN: How do you roll back production?

#101

Post by aasasd »

ransom1538 wrote: post #95
aasasd wrote: post #71

A place I worked at had a symlink pointing to the app directory, and a new version went to a new dir. This allowed us to do atomic deployments: code wasn't replaced while it's being run. A rollback, consequently, meant pointing that symlink to the older version. For the database, during a migration we didn't synchronize code with one version of the db. Database structure was modified to add new fields or tables, and…

"A place I worked at had a symlink pointing to the app directory" This is the way to go. Have your root web directory be a symlink. EG. /var/www/app -> /code_[git_hash]/ You can whip through a thousand vms less than a second with this method. Connect, change the symlink. Other options: Pushing out a new code branch, reverting with git, launching new vms with reverted images, rsync'ing with overwriting -- is slower, a…

> whip through a thousand vms less than a second with this method

I forgot to mention a rather useful quality of this scheme when there are a whole lot of visitors: you can upload the code, switch the link on just a portion of the servers, gawk at the error logs and put the link back if you don't like what you see.

Re: Ask HN: How do you roll back production?

#102

Post by cddotdotslash »

jacobkg wrote: post #67
cddotdotslash wrote: post #24

We actually just implemented something like this. Our entire environment is AWS CodeBuild, CodePipeline, and Lambda-based, but the process would be similar for more traditional environments: 1. Developer creates a PR. To be mergeable, it must pass code review, be based on master, and be up-to-date with master (GitHub recently made this really easy by adding a one-click button to resync master into the PR). 2. Each co…

“GitHub recently made this really easy by adding a one-click button to resync master into the PR” I’m trying to find this feature but my google-fu is failing me. Can you link to an announcement or doc page for this?

I can't find any announcements about it, but you can try it out: open a PR based on master, then push a separate commit (or merge a different PR) to master and return to your original PR. If you scroll down, there will be a box saying the base branch is out of date and a button asking to resync with master. If you click it, they will merge the master branch into your PR.

I'm not sure if there are specific settings required for this to work. For example, we have the master branch protected and require the status checks to pass and the PR to be up to date before it can be merged.

Re: Ask HN: How do you roll back production?

#103

Post by cortesoft »

WrtCdEvrydy wrote: post #82
Scarblac wrote: post #58

Earlier quoted context omitted.

Even the best tests only catch like 50% of the bugs though.

That's not really correct. Properly input space partitioned tests have something like 90 to 95 accuracy if properly written. The issues always come from people not wanting to add sufficient tests.

Does that mean that 1 in 20 deploys breaks, then?

Re: Ask HN: How do you roll back production?

#104

Post by ksajadi »

Since we run on Kubernetes, rolling back the code is a matter of redeploying the older images. Rolling back database changes is more challenging but usually we have “down” scripts as well as up scrips for all dB changes allowing us to roll database changes back too.

We use Cloud 66 Skycap for deployment which gives us a version controlled repository for our Kubernetes configuration files as well as takes care of image tags for each release.

Re: Ask HN: How do you roll back production?

#105

Post by KaiserPro »

WE have two things that could need to be rolled back, the app/api and the dataset.

The App is docker, so we have a tag called app-production, and app-production-1(up to 5) which are all the previous production versions. If anything goes wrong, we can flip over to the last known good version.

We are multi-region, so we don't update all at once.

The dataset is a bit harder. Because its > 100gigs, and for speed purposes it lives on EFS (its lots of 4meg files, and we might need to pull in 60 or so files at once, access time is rubbish using S3) Manually syncing it takes a couple of hours.

To get round this, we have a copy on write system, with "dataset-prod" and "dataset-prod-1" up to 6. Changing the symlink of the top level directory is minimal.

Re: Ask HN: How do you roll back production?

#106

Post by [deleted] »

not_kurt_godel wrote: post #94
aasasd wrote: post #91

Earlier quoted context omitted.

Bahahaha, your tests really aren't ‘best’. Properly, if your code has an ‘if’, you need two tests, for the two branches. Same with every place the outcome may diverge. With this approach, it's basically impossible to botch the code unless something slips your mind while writing both the code and the tests. Otherwise, it's pretty much ‘deploy and go home.’

This is good in theory, and often in practice, but not always. If this practice is followed blindly, unit tests can become too tightly bound to the code to the point where every code change is assumed to require corresponding 1:1 unit test changes and it becomes difficult to distinguish between a broken test that signals broken functionality versus one that simply hasn't been 'synced' yet.

[deleted]

Re: Ask HN: How do you roll back production?

#107

Post by xorcist »

timdorr wrote: post #44
ccmcarey wrote: post #17

Earlier quoted context omitted.

How do you deal with DB migrations?

Migrations should be separated out from other code changes. If you have a rolling deploy process, then you need to make sure your database changes are forwards and backwards compatible. Assuming you've got a CI in place, making the migrations a separate, testable commit will let you do this easily. We did this at my last company with a small GitHub bot and a CODEOWNERS file.

> you need to make sure your database changes are forwards and backwards compatible

That's just another way of saying "you need to make sure you can roll forwards and backwards".

The question was how do you do it. The answer should include the phrase "we test it".

Re: Ask HN: How do you roll back production?

#108

Post by beebs93 »

aasasd wrote: post #101
ransom1538 wrote: post #95

Earlier quoted context omitted.

"A place I worked at had a symlink pointing to the app directory" This is the way to go. Have your root web directory be a symlink. EG. /var/www/app -> /code_[git_hash]/ You can whip through a thousand vms less than a second with this method. Connect, change the symlink. Other options: Pushing out a new code branch, reverting with git, launching new vms with reverted images, rsync'ing with overwriting -- is slower, a…

> whip through a thousand vms less than a second with this method I forgot to mention a rather useful quality of this scheme when there are a whole lot of visitors: you can upload the code, switch the link on just a portion of the servers, gawk at the error logs and put the link back if you don't like what you see.

Yeah, very useful strategy. I've heard it go by various names (e.g. toe-dipping, one-boxing, etc.), but it's always been one of additional methods of helping ensuring safe prod deployments at any company I've worked for.

One downside - depending on your setup - is you may not have an easy way to hit the hosts directly/deterministically via any UIs in case you wanted to do any manual verification/debugging yourself.

Re: Ask HN: How do you roll back production?

#109

Post by sagichmal »

beebs93 wrote: post #108
aasasd wrote: post #101

Earlier quoted context omitted.

> whip through a thousand vms less than a second with this method I forgot to mention a rather useful quality of this scheme when there are a whole lot of visitors: you can upload the code, switch the link on just a portion of the servers, gawk at the error logs and put the link back if you don't like what you see.

Yeah, very useful strategy. I've heard it go by various names (e.g. toe-dipping, one-boxing, etc.), but it's always been one of additional methods of helping ensuring safe prod deployments at any company I've worked for. One downside - depending on your setup - is you may not have an easy way to hit the hosts directly/deterministically via any UIs in case you wanted to do any manual verification/debugging yourself.

> Yeah, very useful strategy. I've heard it go by various names (e.g. toe-dipping, one-boxing, etc.)

Strange terms. Isn't this just a canary?

Re: Ask HN: How do you roll back production?

#110

Post by pojzon »

Shalle135 wrote: post #96
karka91 wrote: post #89

Assuming this is about web dev. Nowadays - flip a toggle in the admin. Deployments and releases are separated. Made a major blunder? In kubernetes world we do "helm rollback". Takes seconds. This allows for a super fast pipeline and a team of 6 devs pushes out like 50 deployments a day. Pre-kubernetes it would be AWS pipeline that would startup servers with old commits. We'd catch most of the stuff in blue/green phas…

Why not canary releases? You can load balance for example 1% of the traffic to the new deployment and see if you experience any issues. If you do - you just change the loadbalancer to use the known good pods.

How you take care of DB updates when using cannary deployments ? For example those which are not backwards compatible ?

Ps. Releases are about building new versions of code packages. Deployments about pushing them out to environments.

Post reply on HN