Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you roll back production?

news.ycombinator.com

Post reply on HN
Print view View original

91–100 of 156 posts

Re: Ask HN: How do you roll back production?

#91

Post by aasasd »

Scarblac wrote: post #58
ISL wrote: post #16

Earlier quoted context omitted.

A test can test for those: "does the code give mathematically correct results?"

Even the best tests only catch like 50% of the bugs though.

Bahahaha, your tests really aren't ‘best’.

Properly, if your code has an ‘if’, you need two tests, for the two branches. Same with every place the outcome may diverge. With this approach, it's basically impossible to botch the code unless something slips your mind while writing both the code and the tests. Otherwise, it's pretty much ‘deploy and go home.’

Re: Ask HN: How do you roll back production?

#92

Post by wy35 »

aasasd wrote: post #71

A place I worked at had a symlink pointing to the app directory, and a new version went to a new dir. This allowed us to do atomic deployments: code wasn't replaced while it's being run. A rollback, consequently, meant pointing that symlink to the older version. For the database, during a migration we didn't synchronize code with one version of the db. Database structure was modified to add new fields or tables, and…

Since you mentioned symlink, does that mean you have each production version of the app on the same machine?

Re: Ask HN: How do you roll back production?

#93

Post by EspadaV9 »

procinct wrote: post #61
timdorr wrote: post #44

Earlier quoted context omitted.

Migrations should be separated out from other code changes. If you have a rolling deploy process, then you need to make sure your database changes are forwards and backwards compatible. Assuming you've got a CI in place, making the migrations a separate, testable commit will let you do this easily. We did this at my last company with a small GitHub bot and a CODEOWNERS file.

Say you were to rename a column, would you achieve this by duplicating the column with the duplicate having the new name?

Yes, this is we've just done recently. Renaming a column required creating a new one with the new name and saving the same data to both columns for a while before dropping old old column a few weeks later.

Re: Ask HN: How do you roll back production?

#94

Post by not_kurt_godel »

aasasd wrote: post #91
Scarblac wrote: post #58

Earlier quoted context omitted.

Even the best tests only catch like 50% of the bugs though.

Bahahaha, your tests really aren't ‘best’. Properly, if your code has an ‘if’, you need two tests, for the two branches. Same with every place the outcome may diverge. With this approach, it's basically impossible to botch the code unless something slips your mind while writing both the code and the tests. Otherwise, it's pretty much ‘deploy and go home.’

This is good in theory, and often in practice, but not always. If this practice is followed blindly, unit tests can become too tightly bound to the code to the point where every code change is assumed to require corresponding 1:1 unit test changes and it becomes difficult to distinguish between a broken test that signals broken functionality versus one that simply hasn't been 'synced' yet.

Re: Ask HN: How do you roll back production?

#95

Post by ransom1538 »

aasasd wrote: post #71

A place I worked at had a symlink pointing to the app directory, and a new version went to a new dir. This allowed us to do atomic deployments: code wasn't replaced while it's being run. A rollback, consequently, meant pointing that symlink to the older version. For the database, during a migration we didn't synchronize code with one version of the db. Database structure was modified to add new fields or tables, and…

"A place I worked at had a symlink pointing to the app directory"

This is the way to go. Have your root web directory be a symlink. EG. /var/www/app -> /code_[git_hash]/ You can whip through a thousand vms less than a second with this method. Connect, change the symlink. Other options: Pushing out a new code branch, reverting with git, launching new vms with reverted images, rsync'ing with overwriting -- is slower, and more dangerous with prod.

"For the database, during a migration"

There is no such thing as a database migration on prod. There is just adding columns. Code should work with new columns added at any point. Altering a column or dropping a column is extremely dangerous.

Re: Ask HN: How do you roll back production?

#96

Post by Shalle135 »

karka91 wrote: post #89

Assuming this is about web dev. Nowadays - flip a toggle in the admin. Deployments and releases are separated. Made a major blunder? In kubernetes world we do "helm rollback". Takes seconds. This allows for a super fast pipeline and a team of 6 devs pushes out like 50 deployments a day. Pre-kubernetes it would be AWS pipeline that would startup servers with old commits. We'd catch most of the stuff in blue/green phas…

Why not canary releases? You can load balance for example 1% of the traffic to the new deployment and see if you experience any issues. If you do - you just change the loadbalancer to use the known good pods.

Re: Ask HN: How do you roll back production?

#97

Post by twunde »

wy35 wrote: post #92
aasasd wrote: post #71

A place I worked at had a symlink pointing to the app directory, and a new version went to a new dir. This allowed us to do atomic deployments: code wasn't replaced while it's being run. A rollback, consequently, meant pointing that symlink to the older version. For the database, during a migration we didn't synchronize code with one version of the db. Database structure was modified to add new fields or tables, and…

Since you mentioned symlink, does that mean you have each production version of the app on the same machine?

Typically yes. This pattern was more common prior to containers and I think built into Ruby's Capistrano. The main benefit is that rollbacks were _quick_ especially compared to kubernetes or ecs

Re: Ask HN: How do you roll back production?

#98

Post by korijn »

karka91 wrote: post #89

Assuming this is about web dev. Nowadays - flip a toggle in the admin. Deployments and releases are separated. Made a major blunder? In kubernetes world we do "helm rollback". Takes seconds. This allows for a super fast pipeline and a team of 6 devs pushes out like 50 deployments a day. Pre-kubernetes it would be AWS pipeline that would startup servers with old commits. We'd catch most of the stuff in blue/green phas…

> Nowadays - flip a toggle in the admin. Deployments and releases are separated.

Would you mind explaining this a little further? How does the separation allow you to flip a switch in the admin?

Re: Ask HN: How do you roll back production?

#99

Post by aasasd »

wy35 wrote: post #92
aasasd wrote: post #71

A place I worked at had a symlink pointing to the app directory, and a new version went to a new dir. This allowed us to do atomic deployments: code wasn't replaced while it's being run. A rollback, consequently, meant pointing that symlink to the older version. For the database, during a migration we didn't synchronize code with one version of the db. Database structure was modified to add new fields or tables, and…

Since you mentioned symlink, does that mean you have each production version of the app on the same machine?

Not ‘each,’ but a few recent versions.

Not sure what you mean by ‘on the same machine,’ though. It's not like we flip server roles between releases. Multiple backend servers all had those several versions of the backend code, and the machines serving the frontend stuff had corresponding versions of that.

Re: Ask HN: How do you roll back production?

#100

Post by MaxGabriel »

For our backend, we deploy it as a nix package on NixOS, so we can atomically rollback the deployed code, as well as any dependencies like system libraries. Right now this requires SSHing into each of our two backend servers and running a command.

If it’s not urgent we’d just revert with a PR though and let the regular deploy process handle it.

The frontend we deploy with Heroku, so we deploy with the rollback button or Heroku CLI. Unfortunately we don’t have something setup where the frontend checks if it’s on the correct version or not, so people will get the bad code until they refresh

Post reply on HN