131–140 of 156 posts
It's just doing another deployment. It doesn't matter what version you are deploying.
That's the whole point.
My teams go into their CI/CD platform and just cherry pick which build they want to release.
I mean it's not rolling back or rolling forward. It's just doing another deployment. It doesn't matter what version you are deploying. That's the whole point. My teams go into their CI/CD platform and just cherry pick which build they want to release.
100% of my rollbacks were like this: * Deploy new code in new VMs. * Route some prod traffic to the new nodes. * Watch the nodes misbehave somehow. * Route 100% of the prod traffic back to old nodes (which nobody tore down). Rollback complete. In the case of normal deployment, 100% of prod traffic would eventually be directed to new modes. After a few hours of everything running smoothly, the old nodes would be spun…
For our backend, we deploy it as a nix package on NixOS, so we can atomically rollback the deployed code, as well as any dependencies like system libraries. Right now this requires SSHing into each of our two backend servers and running a command. If it’s not urgent we’d just revert with a PR though and let the regular deploy process handle it. The frontend we deploy with Heroku, so we deploy with the rollback button…
Earlier quoted context omitted.
Even the best tests only catch like 50% of the bugs though.
Bahahaha, your tests really aren't ‘best’. Properly, if your code has an ‘if’, you need two tests, for the two branches. Same with every place the outcome may diverge. With this approach, it's basically impossible to botch the code unless something slips your mind while writing both the code and the tests. Otherwise, it's pretty much ‘deploy and go home.’
And if the same unit has multiple branches, you need double the tests to cover all paths.
And that doesn't guarantee correctness, and it's only unit tests.
But the 50% was a number I vaguely recall from Code Complete.
A place I worked at had a symlink pointing to the app directory, and a new version went to a new dir. This allowed us to do atomic deployments: code wasn't replaced while it's being run. A rollback, consequently, meant pointing that symlink to the older version. For the database, during a migration we didn't synchronize code with one version of the db. Database structure was modified to add new fields or tables, and…
BTW, copying the entire app to a new directory did begin to take a while after a couple years, and I never got around to solving that. However, I believe it's quite possible to implement COW: hardlink the hell out of the current version, then beat rsync into replacing differing files instead of updating them in-place. The hardlinking part could be done before the deployment even begins—or you could copy the files ins…
We also used to do a symlink method, but proper kubernetes ci/cd setup is so much more better.
I mean it's not rolling back or rolling forward. It's just doing another deployment. It doesn't matter what version you are deploying. That's the whole point. My teams go into their CI/CD platform and just cherry pick which build they want to release.
How do they know which release to cherry pick?
I work for an enterprise. We automated the change control process and integrated it with our CD dashboard. So a quick peak at the dashboard will tell us.
Though depending on the criticality of the app. We may only retain a previous release for a specific time period.
or build a new VM and send it to them.
Not everything is web-based