Ask HN: How do you roll back production?

GitLab CI/CD is amazing. I run a private instance on a VPS with a single worker (probably the simplest setup you can imagine) and it's astoundingly powerful.

I can't say enough great things about it - solutions like Jenkins and Travis CI just feel antiquated and clunky anymore. I always thought it wouldn't really be worth it to run CI/CD on my personal projects due to the complexity inherent in setting these solutions up until I saw the light... I had a coherent "one-click" deploy setup from scratch within an hour with GitLab.

"toxic hellhole", "blame culture" - I don't think we need those dark, marginal extremes to make a point

I also suspect that we're going to agree to disagree here. There are so many nuances, it's impossible to properly communicate most of those without writing a chapter of a book.

Appreciate your points though. Great food for thought right there.

I have multiple strategies because I've got one foot in Docker and one foot in the "old-school" realm of simple web servers.

Code rollbacks are simple as heck - I just keep the previous Docker container(s) up for a potential rollback target, and/or have a symlink cutover strategy for the webservers. I use GitLab CI/CD for the majority of what I do so the SCM is not on the server, it's deployed as artifacts (either a clean tested container and/or .tar.gz). If I need to rollback it's a manual operation for the code but I want to keep it that way because I am a strong believer in not automating edge-cases which is what running rollbacks through your CI/CD pipeline is.

Also for code I've been known to even cut a hot image of the running server just in case something goes _really_ sideways. Never had to use it though, and I will only go this far if I'm making actual changes to the CI/CD pipeline (usually).

The biggest concern for me is database changes. You may think I'm nuts but I have been burnt _sooooo_ bad on this (we were all young and dumb at one time right?)... I have multiple points of "oh %$&%" solutions. The first is good migrations - yeah yeah yell at me if you wish... I run things like Laravel for my API's and their migration rollbacks can take care of simple things. TEST YOUR ROLLBACK MIGRATIONS! The second solution is that I cut an actual readslave for each and every update of the application and then segregate it so that I have a "snapshot" that is at-most 1-2 hours out of date.

Have redundancy to your redundancy is my motto... and although my deployments take a 1-3 hours for big changes (cutting hot images of a running server, building/isolating an independent DB slave, shuffling containers, etc.) I've never had a major "lights out" issue that's lasted more than 1hr.

Exactly this. If your db changes are compatible in both directions then you are safe. It's not hard to achieve. Just make sure new columns have default values and your orm can deal with extra undefined columns and don't drop anything during migrations.

With Kubernetes and deployments simply:

kubectl rollout undo deployment/$DEPLOYMENT