Google App Engine, just set previous deploy to live version (one click)
Ask HN: How do you roll back production?
111–120 of 156 posts
Re: Ask HN: How do you roll back production?
#112We use Elastic Beanstalk so we just deploy whatever application version we'd like. Honestly not the biggest fan of that strategy because there is at least a 5 minute period of time while the new instances are provisioned and healthchecked that you just need to wait for.
When compared to our Fastly deploys which are global in seconds, it leaves me wanting a faster solution.
Re: Ask HN: How do you roll back production?
#113Earlier quoted context omitted.
Yeah, very useful strategy. I've heard it go by various names (e.g. toe-dipping, one-boxing, etc.), but it's always been one of additional methods of helping ensuring safe prod deployments at any company I've worked for. One downside - depending on your setup - is you may not have an easy way to hit the hosts directly/deterministically via any UIs in case you wanted to do any manual verification/debugging yourself.
> Yeah, very useful strategy. I've heard it go by various names (e.g. toe-dipping, one-boxing, etc.) Strange terms. Isn't this just a canary?
Every company I've been with seems to re-invent the terms or swich their definitions slightly.
For me, currently, "canary" means a set of basic automated integration tests that are continually running in production with alarms that feed into a master aggregate "switch". Wether the dedicated canary accounts end up hitting a one-box prod host or real prod host in the end isn't a factor.
The important thing is we incrementally expose our latest code commit to prod hosts via one-boxing to reduce the customer exposure if an acute problem somehow gets past the previous code deploy stages/tests.
Re: Ask HN: How do you roll back production?
#114Earlier quoted context omitted.
Well, highly unlikely that such an error will arise. If it does and you know that you actually need to roll back then most likely something else is wrong. But again, I also saw the other comment about how "Dogmatic" my approach is. I wouldn't say it's dogmatic, idealistic - yes. But not dogmatic. There is a place and time for anything and roll back can STILL be useful when you don't trust the system nor the code base…
Oh please. You never released a bug to production that couldn't be fixed in five minutes? No way. That's such bullshit if you've ever worked in this industry. Legacy systems or not. Even as a fantasy/lie your claim is not believable.
For an obvious typo in a prod config (something like wrong addr:port), sure. But if I think I have an 90% chance of fixing a failure I just introduced (in code I just demonstrated not fully understanding), that's a 10% risk of prolonging a production outage for nothing more than style points. Roll the damn thing back and then fix it with a clear head. That should be a well-practiced reflex. If it ends up being one bad deploy, one rollback, and one healthy deploy all back-to-back, that's totally fine.
In a blue/green world, failing back prod to the cluster with the known-good code should be reflexive. After that maybe you can be lax about whether to roll back or fix forward on the unhealthy cluster (or maybe not, if you only have n=2 clusters).
Re: Ask HN: How do you roll back production?
#115Earlier quoted context omitted.
Even the best tests only catch like 50% of the bugs though.
Bahahaha, your tests really aren't ‘best’. Properly, if your code has an ‘if’, you need two tests, for the two branches. Same with every place the outcome may diverge. With this approach, it's basically impossible to botch the code unless something slips your mind while writing both the code and the tests. Otherwise, it's pretty much ‘deploy and go home.’
> Properly, if your code has an ‘if’, you need two tests, for the two branches. Same with every place the outcome may diverge. With this approach, it's basically impossible to botch the code
if (x % 3 == 0)
println "Fizz"
if (x % 5 == 0)
println "Buzz"
Seriously though, the criterion you mentioned is only one of many of increasing strictness (see e.g. https://en.wikipedia.org/wiki/Code_coverage#Basic_coverage_c...), namely branch coverage. Having branch coverage still says very little - the interaction between different branches can be trivially wrong. And desiring full path coverage immediately leads to combinatorial explosion (and the halting problem, once loops are involved).
> unless something slips your mind while writing both the code and the tests.
That is true for any choice of coverage metric and target.
Re: Ask HN: How do you roll back production?
#116Earlier quoted context omitted.
You didn't really address my points. Its hard to quantify just how "highly unlikely" a failure is, but its your job as a systems designer to build systems that are robust under a wide variety of unlikely failure scenarios. Not having rollbacks results in a system that is extremely problematic in those unlikely scenarios where a quick fix cannot be immediately addressed. Not to mention, rolling forward under such a re…
I'm strictly against rollbacks and I'm strictly for everything continuous. If I need to do a roll back it means that I don't trust the system nor the code base. I will do the roll back but after that there will be a very productive retro about how we can do better to avoid rollbacks in the future (aka what did we learn). But again, as I said, there is a place and time for everything! And there are many variables! Eve…
If a deploy causes an outage, I don't trust that code, and we shouldn't assume we fully understand what's wrong with it. I want the known-good version out ASAP. I don't even want to spend the time to discuss what to do, not until after prod is up.
Re: Ask HN: How do you roll back production?
#117For our backend, we deploy it as a nix package on NixOS, so we can atomically rollback the deployed code, as well as any dependencies like system libraries. Right now this requires SSHing into each of our two backend servers and running a command. If it’s not urgent we’d just revert with a PR though and let the regular deploy process handle it. The frontend we deploy with Heroku, so we deploy with the rollback button…
> For our backend, we deploy it as a nix package on NixOS, so we can atomically rollback the deployed code, as well as any dependencies like system libraries
Same for us, but we use nixpkgs directly over CentOS. Nix is perfect for rollback. It can be done on an entire cluster in seconds.
For the DB, We use schemaless DBs with Devs that care about forward and backward compatibility.
Re: Ask HN: How do you roll back production?
#118Earlier quoted context omitted.
Why not canary releases? You can load balance for example 1% of the traffic to the new deployment and see if you experience any issues. If you do - you just change the loadbalancer to use the known good pods.
How you take care of DB updates when using cannary deployments ? For example those which are not backwards compatible ? Ps. Releases are about building new versions of code packages. Deployments about pushing them out to environments.
Best way is to not write forwards / backwards incompatible database changes.
Re: Ask HN: How do you roll back production?
#119A place I worked at had a symlink pointing to the app directory, and a new version went to a new dir. This allowed us to do atomic deployments: code wasn't replaced while it's being run. A rollback, consequently, meant pointing that symlink to the older version. For the database, during a migration we didn't synchronize code with one version of the db. Database structure was modified to add new fields or tables, and…
"A place I worked at had a symlink pointing to the app directory" This is the way to go. Have your root web directory be a symlink. EG. /var/www/app -> /code_[git_hash]/ You can whip through a thousand vms less than a second with this method. Connect, change the symlink. Other options: Pushing out a new code branch, reverting with git, launching new vms with reverted images, rsync'ing with overwriting -- is slower, a…
>There is no such thing as a database migration on prod. There is just adding columns.
We have a DB schema that we not so affectionately refer to as the Standard Oil Octopus because of this methodology applied over ~20yr.
I agree with you in the general case but eventually hard cuts have to be made or you will perpetuate the existence of all sorts of legacy spaghetti (not necessarily in the DB, but in all the other things that use the DB). Like everything else there's a balance to be struck.
Re: Ask HN: How do you roll back production?
#120A place I worked at had a symlink pointing to the app directory, and a new version went to a new dir. This allowed us to do atomic deployments: code wasn't replaced while it's being run. A rollback, consequently, meant pointing that symlink to the older version. For the database, during a migration we didn't synchronize code with one version of the db. Database structure was modified to add new fields or tables, and…
"A place I worked at had a symlink pointing to the app directory" This is the way to go. Have your root web directory be a symlink. EG. /var/www/app -> /code_[git_hash]/ You can whip through a thousand vms less than a second with this method. Connect, change the symlink. Other options: Pushing out a new code branch, reverting with git, launching new vms with reverted images, rsync'ing with overwriting -- is slower, a…
> There is no such thing as a database migration on prod. There is just adding columns. Code should work with new columns added at any point.
If you can't alter a column, how do you prevent your database slowly rotting in terms of its design integrity?