Rollback the master branch and deploy that again.
Similar to "clicking the button in heroku"
Ask HN: How do you roll back production?
81–90 of 156 posts
Re: Ask HN: How do you roll back production?
#82Earlier quoted context omitted.
A test can test for those: "does the code give mathematically correct results?"
Even the best tests only catch like 50% of the bugs though.
That's not really correct.
Properly input space partitioned tests have something like 90 to 95 accuracy if properly written.
The issues always come from people not wanting to add sufficient tests.
Re: Ask HN: How do you roll back production?
#83Depends on the issues.
1) Issues that cause a complete failure to start containers will fail healthchecks and are auto rolled back in our new CI/CD flow.
2) Issues that are more subtle are manually rolled to one back hash until it goes away (then we create a revert branch from that diff between HEAD and WORKING).
Re: Ask HN: How do you roll back production?
#84Earlier quoted context omitted.
What about logical errors? math.pow(2, 4) vs math.pow(4, 2)
Semantic errors? I don't think a unit test will be able to distinguish 16 from 16.
Imagine this: the spec is Math.pow(x,y). It was implemented as Math.pow(y, x). The test was written with x=2, y=4. It is accidentally correct. This is why any unit tests dealing with absolute or enum values needs to run multiple iterations.
Re: Ask HN: How do you roll back production?
#85Earlier quoted context omitted.
When do you drop the old e.g. columns and tables that are no longer in use though? You schedule it for say a week later when you're sure you won't need to roll back that far?
Correct. We 'mark' no longer used database objects for removal at some future date. And mark is really just add a ticket to be completed in the future.
How long does it stick around before final removal, typically?
Re: Ask HN: How do you roll back production?
#86Earlier quoted context omitted.
A test can test for those: "does the code give mathematically correct results?"
I meant pow as an example to point out that there are logical errors in code that you cannot catch with unit tests.
those kinds of errors should be caught a little higher in the hierarchy... some kind of feature or integration test.
Re: Ask HN: How do you roll back production?
#87Earlier quoted context omitted.
Until the fix is identified, can't one 'march forward' by rolling back recent changes to a known-good state?
Nope, you are "pretending" that a step back is a step forward (which is not true). I never had to roll back anything within last couple years and very happy about that. Also note, that roll back was a valid strategy back in the day and still can be useful tool in your garage of tools. It can be useful when, for example, dealing with complex legacy systems that were created decades ago, or complex systems developed by…
> Nope, you are "pretending" that a step back is a step forward (which is not true)
If the business is losing $BIG_BUCKS per minute of downtime, it is most definitely a step forward.
Re: Ask HN: How do you roll back production?
#88Earlier quoted context omitted.
When do you drop the old e.g. columns and tables that are no longer in use though? You schedule it for say a week later when you're sure you won't need to roll back that far?
Correct. We 'mark' no longer used database objects for removal at some future date. And mark is really just add a ticket to be completed in the future.
Do you have a method to rollback database object removals if they cause problems?
Re: Ask HN: How do you roll back production?
#89Assuming this is about web dev.
Nowadays - flip a toggle in the admin. Deployments and releases are separated.
Made a major blunder? In kubernetes world we do "helm rollback". Takes seconds. This allows for a super fast pipeline and a team of 6 devs pushes out like 50 deployments a day.
Pre-kubernetes it would be AWS pipeline that would startup servers with old commits. We'd catch most of the stuff in blue/green phase, though. Same team, maybe 10 deployments a day but I think this was still pretty good for a monolith.
Pre-aws we used deployment tools like capistrano. Most of the tools in this category have multiple releases on the servers and a symlink to the live one. If you make mistake - run a command to delete the symlink, ln -s old release, restart web server. Even though this is the fastest rollback of the bunch the ecosystem was still young and we'd do 0-2 releases a day.
Re: Ask HN: How do you roll back production?
#90interesting question. we roll forward and thus far never ran into the situation that that wasn't possible in a reasonable amount of time. nevertheless i've wondered more than once what would happen if we run into such a situation and there's a substantial database migration in the process (i.e. with table drops). curious to learn what the different strategies are on that point: do you put your table contents in the d…
Generally a good strategy for dropping columns or tables is to rename them instead (e.g. `table_deprecated`).
If things look stable after whatever time you deem necessary, you can write a second migration to actually drop them.
If you run into issues, your down migration simply undoes the rename.