Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you roll back production?

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 156 posts

Re: Ask HN: How do you roll back production?

#61

Post by procinct »

timdorr wrote: post #44
ccmcarey wrote: post #17

Earlier quoted context omitted.

How do you deal with DB migrations?

Migrations should be separated out from other code changes. If you have a rolling deploy process, then you need to make sure your database changes are forwards and backwards compatible. Assuming you've got a CI in place, making the migrations a separate, testable commit will let you do this easily. We did this at my last company with a small GitHub bot and a CODEOWNERS file.

Say you were to rename a column, would you achieve this by duplicating the column with the duplicate having the new name?

Re: Ask HN: How do you roll back production?

#63

Post by seanwilson »

kyriakos wrote: post #57
timdorr wrote: post #44

Earlier quoted context omitted.

Migrations should be separated out from other code changes. If you have a rolling deploy process, then you need to make sure your database changes are forwards and backwards compatible. Assuming you've got a CI in place, making the migrations a separate, testable commit will let you do this easily. We did this at my last company with a small GitHub bot and a CODEOWNERS file.

Exactly this. If your db changes are compatible in both directions then you are safe. It's not hard to achieve. Just make sure new columns have default values and your orm can deal with extra undefined columns and don't drop anything during migrations.

When do you drop the old e.g. columns and tables that are no longer in use though? You schedule it for say a week later when you're sure you won't need to roll back that far?

Re: Ask HN: How do you roll back production?

#65

Post by gingerlime »

peterangular wrote: post #51
jekrb wrote: post #46

At the agency I used to work for, we used GitLab CI/CD. We were able to do a manual rollback for each deployment from the GitLab UI. https://docs.gitlab.com/ee/ci/environments.html#retrying-and... Disclaimer: I work at GitLab now, but my old agency was also using GitLab and their CI/CD offering for client projects for a couple years while I was there. At that agency they have even open sourced their GitLab CI configs…

GitLab CI/CD is amazing. I run a private instance on a VPS with a single worker (probably the simplest setup you can imagine) and it's astoundingly powerful. I can't say enough great things about it - solutions like Jenkins and Travis CI just feel antiquated and clunky anymore. I always thought it wouldn't really be worth it to run CI/CD on my personal projects due to the complexity inherent in setting these solution…

I'm curious about the difference. Jenkins, ok. Sure. But in what way are Travis/CircleCI/Semaphore vastly different from Gitlab CI/CD? honest question.

Re: Ask HN: How do you roll back production?

#66

Post by matwood »

seanwilson wrote: post #63
kyriakos wrote: post #57

Earlier quoted context omitted.

Exactly this. If your db changes are compatible in both directions then you are safe. It's not hard to achieve. Just make sure new columns have default values and your orm can deal with extra undefined columns and don't drop anything during migrations.

When do you drop the old e.g. columns and tables that are no longer in use though? You schedule it for say a week later when you're sure you won't need to roll back that far?

Correct. We 'mark' no longer used database objects for removal at some future date. And mark is really just add a ticket to be completed in the future.

Re: Ask HN: How do you roll back production?

#67

Post by jacobkg »

cddotdotslash wrote: post #24

We actually just implemented something like this. Our entire environment is AWS CodeBuild, CodePipeline, and Lambda-based, but the process would be similar for more traditional environments: 1. Developer creates a PR. To be mergeable, it must pass code review, be based on master, and be up-to-date with master (GitHub recently made this really easy by adding a one-click button to resync master into the PR). 2. Each co…

“GitHub recently made this really easy by adding a one-click button to resync master into the PR”

I’m trying to find this feature but my google-fu is failing me. Can you link to an announcement or doc page for this?

Re: Ask HN: How do you roll back production?

#68

Post by NicoJuicy »

timdorr wrote: post #44
ccmcarey wrote: post #17

Earlier quoted context omitted.

How do you deal with DB migrations?

Migrations should be separated out from other code changes. If you have a rolling deploy process, then you need to make sure your database changes are forwards and backwards compatible. Assuming you've got a CI in place, making the migrations a separate, testable commit will let you do this easily. We did this at my last company with a small GitHub bot and a CODEOWNERS file.

That's not always possible in an easy way. It leads to more problems than it solves as soon as there are >1 developers. Your code is dependant on a current state of the database and you should factor that in with your deployments.

My deployment have a optional Target on the migration for rollbacks and I have never had any problems.

Depends on the tech you use ofc. But for. Net the entity framework make it easy.

Your comment is only used as a excuse for imperfect deployments and creates too much additional problems.

You should really do it like this, so the code decides how the connected database should.

Otherwhise the human overhead to execute the correct scripts will have a problem sooner or later ( eg. If a separate team/person handjes deployments). It's also vague to know if someone else has configured everything correct ( at my current job, they put all the scripts in a separate branch. One application connects to another DB and ofc the DB deployments where not configured yet).

Urgh

Re: Ask HN: How do you roll back production?

#69

Post by mnm1 »

rinchik wrote: post #34
gfodor wrote: post #27

Earlier quoted context omitted.

Shorter releases can help with reducing the difficulty of immediately addressing problems, but it's an error to equate the reduction of risk as an elimination of risk. There are always going to be failure modes that require extensive time to diagnose and debug, even with small changes being made. Additionally, you want that diagnostic phase to happen without time pressure. If you do not have a sane rollback mechanism…

Well, highly unlikely that such an error will arise. If it does and you know that you actually need to roll back then most likely something else is wrong. But again, I also saw the other comment about how "Dogmatic" my approach is. I wouldn't say it's dogmatic, idealistic - yes. But not dogmatic. There is a place and time for anything and roll back can STILL be useful when you don't trust the system nor the code base…

Oh please. You never released a bug to production that couldn't be fixed in five minutes? No way. That's such bullshit if you've ever worked in this industry. Legacy systems or not. Even as a fantasy/lie your claim is not believable.

Re: Ask HN: How do you roll back production?

#70

Post by ouaiueoaieaeoiu »

StreamBright wrote: post #8
httpsterio wrote: post #5

Earlier quoted context omitted.

In our case its quite simple. All commits should be tested and if we need to roll back, it means either that tests have failed and it was pushed nonetheless, tests were missing or tests didn't catch the issue. In the first two cases, we revert to the commit where untested or failed code was introduced to the master. This basically never happens. In the third case, you need to do some debugging and try to figure out w…

What about logical errors? math.pow(2, 4) vs math.pow(4, 2)

Semantic errors?

I don't think a unit test will be able to distinguish 16 from 16.

Post reply on HN