Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you roll back production?

news.ycombinator.com

Post reply on HN
Print view View original

121–130 of 156 posts

Re: Ask HN: How do you roll back production?

#121

Post by ransom1538 »

zmmmmm wrote: post #120
ransom1538 wrote: post #95

Earlier quoted context omitted.

"A place I worked at had a symlink pointing to the app directory" This is the way to go. Have your root web directory be a symlink. EG. /var/www/app -> /code_[git_hash]/ You can whip through a thousand vms less than a second with this method. Connect, change the symlink. Other options: Pushing out a new code branch, reverting with git, launching new vms with reverted images, rsync'ing with overwriting -- is slower, a…

> There is no such thing as a database migration on prod. There is just adding columns. Code should work with new columns added at any point. If you can't alter a column, how do you prevent your database slowly rotting in terms of its design integrity?

Sure. After the code has been live with the new column for two (?) weeks - drop the column you don't need. Alters = add a new column, weeks later delete the old column.

Re: Ask HN: How do you roll back production?

#122

Post by [deleted] »

dsfyu404ed wrote: post #119
ransom1538 wrote: post #95

Earlier quoted context omitted.

"A place I worked at had a symlink pointing to the app directory" This is the way to go. Have your root web directory be a symlink. EG. /var/www/app -> /code_[git_hash]/ You can whip through a thousand vms less than a second with this method. Connect, change the symlink. Other options: Pushing out a new code branch, reverting with git, launching new vms with reverted images, rsync'ing with overwriting -- is slower, a…

>There is no such thing as a database migration on prod. There is just adding columns. We have a DB schema that we not so affectionately refer to as the Standard Oil Octopus because of this methodology applied over ~20yr. I agree with you in the general case but eventually hard cuts have to be made or you will perpetuate the existence of all sorts of legacy spaghetti (not necessarily in the DB, but in all the other t…

[deleted]

Re: Ask HN: How do you roll back production?

#123

Post by ransom1538 »

dsfyu404ed wrote: post #119
ransom1538 wrote: post #95

Earlier quoted context omitted.

"A place I worked at had a symlink pointing to the app directory" This is the way to go. Have your root web directory be a symlink. EG. /var/www/app -> /code_[git_hash]/ You can whip through a thousand vms less than a second with this method. Connect, change the symlink. Other options: Pushing out a new code branch, reverting with git, launching new vms with reverted images, rsync'ing with overwriting -- is slower, a…

>There is no such thing as a database migration on prod. There is just adding columns. We have a DB schema that we not so affectionately refer to as the Standard Oil Octopus because of this methodology applied over ~20yr. I agree with you in the general case but eventually hard cuts have to be made or you will perpetuate the existence of all sorts of legacy spaghetti (not necessarily in the DB, but in all the other t…

Yeah! I wasn't trying to suggest keep columns that are not used for years. I was trying to suggest during that release cycle don't alter or drop columns.

Re: Ask HN: How do you roll back production?

#124

Post by jcampbell1 »

seanwilson wrote: post #88
matwood wrote: post #66

Earlier quoted context omitted.

Correct. We 'mark' no longer used database objects for removal at some future date. And mark is really just add a ticket to be completed in the future.

Do you have a method to rollback database object removals if they cause problems?

Assuming one step per day to rename a column: create new column, update code to write to new and old, update code to read from new, update code only write to new, rename old to something like xxx_del_pending, delete old column.

This process lets you validate and rollback/recover if needed.

Re: Ask HN: How do you roll back production?

#125

Post by aasasd »

MauranKilom wrote: post #115
aasasd wrote: post #91

Earlier quoted context omitted.

Bahahaha, your tests really aren't ‘best’. Properly, if your code has an ‘if’, you need two tests, for the two branches. Same with every place the outcome may diverge. With this approach, it's basically impossible to botch the code unless something slips your mind while writing both the code and the tests. Otherwise, it's pretty much ‘deploy and go home.’

> Properly, if your code has an ‘if’, you need two tests, for the two branches. Same with every place the outcome may diverge. With this approach, it's basically impossible to botch the code if (x % 3 == 0) println "Fizz" if (x % 5 == 0) println "Buzz" There, solved it! And it works fine in all cases you told me to test (e.g. 3, 4, and 5), which means it's impossible I botched anything! Surely I nailed this interview…

> all cases you told me to test (e.g. 3, 4, and 5)! Surely I nailed this interview

Well, if you don't see other points where the input→output mapping can diverge (and rely on the client's requirements for that?), then no, you didn't nail it.

Re: Ask HN: How do you roll back production?

#126

Post by aasasd »

aasasd wrote: post #71

A place I worked at had a symlink pointing to the app directory, and a new version went to a new dir. This allowed us to do atomic deployments: code wasn't replaced while it's being run. A rollback, consequently, meant pointing that symlink to the older version. For the database, during a migration we didn't synchronize code with one version of the db. Database structure was modified to add new fields or tables, and…

BTW, copying the entire app to a new directory did begin to take a while after a couple years, and I never got around to solving that. However, I believe it's quite possible to implement COW: hardlink the hell out of the current version, then beat rsync into replacing differing files instead of updating them in-place. The hardlinking part could be done before the deployment even begins—or you could copy the files instead of hardlinking to avoid altering them accidentally.

Or could probably just use overlayfs or something like that.

Note that, while containers can give you COW, they shouldn't really be necessary for the files of the app. And with php-fpm, changing the app dir is faster than restarting containers: it's actually done via changing the Nginx config.

Re: Ask HN: How do you roll back production?

#127

Post by rgoulter »

korijn wrote: post #98
karka91 wrote: post #89

Assuming this is about web dev. Nowadays - flip a toggle in the admin. Deployments and releases are separated. Made a major blunder? In kubernetes world we do "helm rollback". Takes seconds. This allows for a super fast pipeline and a team of 6 devs pushes out like 50 deployments a day. Pre-kubernetes it would be AWS pipeline that would startup servers with old commits. We'd catch most of the stuff in blue/green phas…

> Nowadays - flip a toggle in the admin. Deployments and releases are separated. Would you mind explaining this a little further? How does the separation allow you to flip a switch in the admin?

AFAIU this is making use of feature flags.

Uploading new code to a server where the new-code is behind a disabled feature flag means the program hasn't changed. The feature flag can be enabled when it's suitable. (This could even be 'rolled out' to only a subset of users; e.g. testers, internal, 10% of users, etc.)

deploy = put the code on the server, release = enable the feature flags

Re: Ask HN: How do you roll back production?

#128

Post by pedalpete »

mooreds wrote: post #3
robbya wrote: post #2

Specifically, I tell Jenkins to deploy the commit hash that was last known good. Jenkins just deploys, and doesn't really know that it's a "roll back." Generally, going back to a known clean state should be easier, safer and relatively quick (DNS flip is fast, redeploy of old code is fast if your automation works well). In some cases changes to your data may make rolling back cause even more problems. I've seen that…

How do you find the commit hash that is the last known good? Looking through jenkins release logs, asking someone, something else?

We create a release branch for each release so we know we can always go to a previous branch which was safe.

Re: Ask HN: How do you roll back production?

#129

Post by WrtCdEvrydy »

cortesoft wrote: post #103
WrtCdEvrydy wrote: post #82

Earlier quoted context omitted.

That's not really correct. Properly input space partitioned tests have something like 90 to 95 accuracy if properly written. The issues always come from people not wanting to add sufficient tests.

Does that mean that 1 in 20 deploys breaks, then?

1 in 20 deploys has a false negative (IE, something went out without working).

Continous Integration is mostly about regression, so the new tests have less value than running old tests.

We tend to fix issues in production (bugs) by mandating a unit test with the failing issue to be written as part of the fix.

Re: Ask HN: How do you roll back production?

#130

Post by nine_k »

100% of my rollbacks were like this:

* Deploy new code in new VMs.

* Route some prod traffic to the new nodes.

* Watch the nodes misbehave somehow.

* Route 100% of the prod traffic back to old nodes (which nobody tore down).

Rollback complete.

In the case of normal deployment, 100% of prod traffic would eventually be directed to new modes. After a few hours of everything running smoothly, the old nodes would be spun down.

Post reply on HN