Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you roll back production?

news.ycombinator.com

Post reply on HN
Print view View original

71–80 of 156 posts

Re: Ask HN: How do you roll back production?

#71

Post by aasasd »

A place I worked at had a symlink pointing to the app directory, and a new version went to a new dir. This allowed us to do atomic deployments: code wasn't replaced while it's being run. A rollback, consequently, meant pointing that symlink to the older version.

For the database, during a migration we didn't synchronize code with one version of the db. Database structure was modified to add new fields or tables, and the data was migrated, all while the site was online. The code expected to find either version of the db, usually signaled by a flag for the shard. If the changes were too difficult to do online, relatively small shards of the db were put on maintenance. Errors were normally caught after a migration of one shard, so switching it back wasn't too painful. Database operations for the migrations were a mix of automatic updates to the list of table fields, and data-moving queries written by hand in the migration scripts—the db structure didn't really look like your regular ORM anyway.

This approach served us quite well, for about five years that I was there—with a lot of visitors and data, dozens of servers and multiple deployments a day.

Re: Ask HN: How do you roll back production?

#72

Post by MetalGuru »

EliRivers wrote: post #6

Tell people we need to roll back, clone the repo to my hard drive, open up git, undo the commit that merged the bad code in, push it. All done. "Production"? Does that mean something that goes to the customers? Very few of our customers keep up with releases so it's generally not a big deal. We can have a release version sitting around for weeks before any customer actually installs it; some customers are happy with…

Yeah, these strategies are really only critical to client-server architectures where newly deployed code is immediately used by customers.

Re: Ask HN: How do you roll back production?

#73

Post by peterangular »

gingerlime wrote: post #65
peterangular wrote: post #51

Earlier quoted context omitted.

GitLab CI/CD is amazing. I run a private instance on a VPS with a single worker (probably the simplest setup you can imagine) and it's astoundingly powerful. I can't say enough great things about it - solutions like Jenkins and Travis CI just feel antiquated and clunky anymore. I always thought it wouldn't really be worth it to run CI/CD on my personal projects due to the complexity inherent in setting these solution…

I'm curious about the difference. Jenkins, ok. Sure. But in what way are Travis/CircleCI/Semaphore vastly different from Gitlab CI/CD? honest question.

Right on. GitLab has everything right there for me from SCM, to it's own Docker container registry, to static site hosting capabilities (think generated documentation), issues management, etc etc. I've tested the setup of many of them and GitLab was the quickest/easiest/most configurable.

What GitLab gets right is having TONS of enterprise-quality solutions available to you in _one place_... for 100% free as their community offering is AMAZING. That's insanely valuable to me as a startup engineer who doesn't have the time to run 4-5 disparate solutions that are difficult to integrate in a secure/simple way.

Having one solution for the above list has been a "game changer" to me because I've got one monolith piece of software to keep updated/manage vs. stringing a whole bunch of solutions together - and I say "monolith" in a 100% positive context =)

Then there's just the speed issue... I did DevOps at a huge mega-corp not long ago and the expectation of "major things to get done" was 3-4 things a week. Now that I'm doing my own startup my expectation on my self is 3-4 major things _in a day_. GitLab is the only tooling that I can imagine keeping up with me with near-zero BS, and because of that I'm a _huge_ brand advocate for them! (Not directly affiliated, just a passionate user!)

All-in-all I understand people have different tools and that's totally cool, but I did a lot to test out different CI/CD tooling and GitLab was amazingly simple, secure, and quick to setup.

Check out for a solid feature comparison: https://about.gitlab.com/devops-tools/travis-ci-vs-gitlab.ht...

Re: Ask HN: How do you roll back production?

#74

Post by atombender »

ccmcarey wrote: post #17
robbya wrote: post #2

Specifically, I tell Jenkins to deploy the commit hash that was last known good. Jenkins just deploys, and doesn't really know that it's a "roll back." Generally, going back to a known clean state should be easier, safer and relatively quick (DNS flip is fast, redeploy of old code is fast if your automation works well). In some cases changes to your data may make rolling back cause even more problems. I've seen that…

How do you deal with DB migrations?

For us, database migrations are a separate step, and we make sure to know whether a migration results in breakage.

If it does, we take the app down first (with a maintenance message) so that errors won't bubble up to the users and APIs.

If we need to roll back a deploy that has migrations, we just roll back the migration. Every migration requires a "down" step that reverses itself. Our Go apps use Goose [1], but there are many other solutions.

[1] https://github.com/pressly/goose

Re: Ask HN: How do you roll back production?

#75

Post by msbarnett »

mooreds wrote: post #3
robbya wrote: post #2

Specifically, I tell Jenkins to deploy the commit hash that was last known good. Jenkins just deploys, and doesn't really know that it's a "roll back." Generally, going back to a known clean state should be easier, safer and relatively quick (DNS flip is fast, redeploy of old code is fast if your automation works well). In some cases changes to your data may make rolling back cause even more problems. I've seen that…

How do you find the commit hash that is the last known good? Looking through jenkins release logs, asking someone, something else?

Tag your released commits in git. Then just ask your deployment infrastructure to deploy whatever tag was in production immediately prior to the failed deployment.

Re: Ask HN: How do you roll back production?

#79

Post by ericol »

We have a rather simple app that we manage with Github. When a Pr is merged into our main repo's master branch it gets automatically deployed into production.

whenever we need to roll back something we just use the corresponding Github feature to revert a merge, and that is automatically shoved into production using GH hooks and stuff.

Again, we have a rather easy and ancient deploy system, and it just works.

We do several updates a week if needed. We try to avoid late Friday afternoon merges, but with a couple alerts here and there (Mostly, New Relic) we have a good coverage to find out about problems.

Post reply on HN