Ask HN: What CI/CD server/service do you use? • Hacker News Board

Re: Ask HN: What CI/CD server/service do you use?

#21

Post by monty5811 » Thu, Jan 26, 2017, 9:42 AM UTC

Semaphore [0] is great. It was the only place where I could get selenium tests to work reliably.

Re: Ask HN: What CI/CD server/service do you use?

#22

Post by hackerboos » Thu, Jan 26, 2017, 1:42 PM UTC

trcollinson wrote: post #17

jazoom wrote: post #14

Earlier quoted context omitted.

Do you use it for building docker images? I can't work out how to tag my images with versions. They all become "latest". Do I have to change the gitlab config file every push?

I came to second Gitlab CI. I build all of my docker images in Gitlab CI and tag them. The trick is to use the gitlab ci environment variables (I tag with with the git tag or the git branch name depending using this method). If you would like to get a bit more help, feel free to email me. My username here at gmail.

What do you with your docker images after they are build? This is the one thing missing from my CD pipeline.

Re: Ask HN: What CI/CD server/service do you use?

#23

Post by johnnycarcin » Thu, Jan 26, 2017, 3:31 PM UTC

i386 wrote: post #16

johnnycarcin wrote: post #13

After being a Jenkins person for years I recently started playing around with Visual Studio Team Services, Microsoft's hosted TFS product: https://www.visualstudio.com/team-services/ . You don't have to use TFS as the backend as git is supported and all of your build and release stuff is under the same roof. They have added a lot of "tasks" for most of the popular languages but worst case is you can always run shell…

Have you taken Jenkins Blue Ocean [1] and Declarative Pipeline [2] for a spin since moving to TFS? We are quickly modernising everything from the UI to the way you manage your Pipelines in version control. Would be very interested in getting your thoughts on how we stack up now vs your past experience :) [1] https://jenkins.io/projects/blueocean/ [2] https://github.com/jenkinsci/pipeline-model-definition-plugi...

I haven't yet but I did look at it awhile back. I'll try and set aside some time to spin it up and take it for a test drive, thanks for the reminder!

Re: Ask HN: What CI/CD server/service do you use?

#24

Post by accraze » Thu, Jan 26, 2017, 3:48 PM UTC

I've been using GoCD for the past couple of months and I'm really liking it so far. Its very flexible and relatively straight-forward to setup on your own VM. The plugins work similar to the way Jenkin's does it. Also the UI is very clean.

Re: Ask HN: What CI/CD server/service do you use?

#25

Post by trcollinson » Thu, Jan 26, 2017, 5:44 PM UTC

jazoom wrote: post #19

trcollinson wrote: post #17

Earlier quoted context omitted.

I came to second Gitlab CI. I build all of my docker images in Gitlab CI and tag them. The trick is to use the gitlab ci environment variables (I tag with with the git tag or the git branch name depending using this method). If you would like to get a bit more help, feel free to email me. My username here at gmail.

That's very kind of you. Thanks. I think I understand how you do it. You just tag the git commit and get that value in the build by using $CI_BUILD_TAG? The description for that variable says it's only available when building a tag. Does that just mean the commit needs to be tagged before a push? Update: I just found this, which is probably exactly what I want to do: https://github.com/gitlabhq/gitlab-ci/issues/637 U…

So yes, you got it! They use the word "tag" in two totally different ways which is confusing.

When you create a gitlab-ci runner you can "tag" the runner. Basically, this is like a named server. For instance, I have a general gitlab-ci runner which I use to run most of my builds. However, I found that I need a special runner just for building docker images. So most of my jobs in my gitlab-ci are marked with:

  tags:
   - general

Where as my docker build job is marked with:

  tags:
   - docker-builder

This basically pushes the job to a specific runner and has nothing to do at all with the git ref (which could be a branch name or a tag or whatnot).

Now when do I build docker containers? I actually don't use "git ref tags" (i.e. git tag v.1.16.001 or something). I have a development branch, a staging branch, and a production branch which tie to my environments (dev, staging, and production). So my job runner looks something like this:

  stages:
   - test
   - build
   - deploy
  
  docker_build:
    stage: build
    script:
     - do some stuff
     - docker tag container-name:$CI_BUILD_REF_NAME $AWS_ACCOUNT_ID.$AWS_ACCOUNT_REGION.amazonaws.com/container-name:$CI_BUILD_REF_NAME
     - docker push $AWS_ACCOUNT_ID.$AWS_ACCOUNT_REGION.amazonaws.com/container-name:$CI_BUILD_REF_NAME
    tags:
     - docker-builder
    only:
     - development
     - staging
     - master

So basically once the tests pass, and IF the branch is development, staging, or master, it will build the docker container, using the CI_BUILD_REF_NAME as the tag and then push that to my private aws container storage system. The AWS_ACCOUNT_ID and REGION are both environment variables I have injected previously. If that succeeds it moves to the deploy stage.

Re: Ask HN: What CI/CD server/service do you use?

#26

Post by trcollinson » Thu, Jan 26, 2017, 5:55 PM UTC

hackerboos wrote: post #22

trcollinson wrote: post #17

Earlier quoted context omitted.

I came to second Gitlab CI. I build all of my docker images in Gitlab CI and tag them. The trick is to use the gitlab ci environment variables (I tag with with the git tag or the git branch name depending using this method). If you would like to get a bit more help, feel free to email me. My username here at gmail.

What do you with your docker images after they are build? This is the one thing missing from my CD pipeline.

Good question! So first I build the docker container with a Dockerfile I keep at the root of my project. I also have an aws private docker repo. (Read more about it here: https://aws.amazon.com/ecr/). So in my gitlab-ci I have a statement like the following:

  - docker build -t container-name:$CI_BUILD_REF_NAME .
  - docker tag container-name:$CI_BUILD_REF_NAME $AWS_ACCOUNT_ID.$AWS_ACCOUNT_REGION.amazonaws.com/container-name:$CI_BUILD_REF_NAME
  - docker push $AWS_ACCOUNT_ID.$AWS_ACCOUNT_REGION.amazonaws.com/container-name:$CI_BUILD_REF_NAME

This builds my container, tags it, and pushes it to my ec2 container registry. I do most of my deployments using AWS ECS (EC2 Container Service, you can read more about it here: https://aws.amazon.com/ecs/)

My secret sauce is in the deployment stage of my ci. I basically run a command which deploys to a ECS environment. This is a custom script which is part of my overall AWS process. I have scripted out the creation of everything on AWS (from building the environment, spinning up EC2's, security and security groups, auto scaling groups, container registry, deploying jobs into an ECS). In the current project I am in we are using Ruby so the command is just a rake task and looks like:

  stages:
   - test
   - build
   - deploy
  
  docker_deploy:
    stage: deploy
    script:
     - do some stuff
     - rake awsenv:redeploy[$CI_BUILD_REF_NAME]
    tags:
     - general
    only:
     - development
     - staging
     - master

The rake task logs into the awsapi securely, runs some commands, verifies the work, and calls the task complete in gitlab-ci. If you have any further questions, feel free to email me. It's my username at gmail.

Re: Ask HN: What CI/CD server/service do you use?

#27

Post by hackerboos » Thu, Jan 26, 2017, 8:03 PM UTC

trcollinson wrote: post #26

hackerboos wrote: post #22

Earlier quoted context omitted.

What do you with your docker images after they are build? This is the one thing missing from my CD pipeline.

Good question! So first I build the docker container with a Dockerfile I keep at the root of my project. I also have an aws private docker repo. (Read more about it here: https://aws.amazon.com/ecr/ ). So in my gitlab-ci I have a statement like the following: - docker build -t container-name:$CI_BUILD_REF_NAME . - docker tag container-name:$CI_BUILD_REF_NAME $AWS_ACCOUNT_ID.$AWS_ACCOUNT_REGION.amazonaws.com/container…

Thank you for giving such a comprehensive answer.

Re: Ask HN: What CI/CD server/service do you use?

#28

Post by jazoom » Thu, Jan 26, 2017, 8:38 PM UTC

trcollinson wrote: post #25

jazoom wrote: post #19

Earlier quoted context omitted.

That's very kind of you. Thanks. I think I understand how you do it. You just tag the git commit and get that value in the build by using $CI_BUILD_TAG? The description for that variable says it's only available when building a tag. Does that just mean the commit needs to be tagged before a push? Update: I just found this, which is probably exactly what I want to do: https://github.com/gitlabhq/gitlab-ci/issues/637 U…

So yes, you got it! They use the word "tag" in two totally different ways which is confusing. When you create a gitlab-ci runner you can "tag" the runner. Basically, this is like a named server. For instance, I have a general gitlab-ci runner which I use to run most of my builds. However, I found that I need a special runner just for building docker images. So most of my jobs in my gitlab-ci are marked with: tags: -…

Thanks for taking the time to share your experience. What's the difference with your docker runner? I've been building docker images just fine without specifying a runner.

Re: Ask HN: What CI/CD server/service do you use?

#29

Post by trcollinson » Thu, Jan 26, 2017, 9:06 PM UTC

jazoom wrote: post #28

trcollinson wrote: post #25

Earlier quoted context omitted.

So yes, you got it! They use the word "tag" in two totally different ways which is confusing. When you create a gitlab-ci runner you can "tag" the runner. Basically, this is like a named server. For instance, I have a general gitlab-ci runner which I use to run most of my builds. However, I found that I need a special runner just for building docker images. So most of my jobs in my gitlab-ci are marked with: tags: -…

Thanks for taking the time to share your experience. What's the difference with your docker runner? I've been building docker images just fine without specifying a runner.

I run most of my runners as Docker runners and I was having issues with a Docker runner building a docker image. So I swapped to a different method. But I haven't tried it in a while. I might be able to use the same Docker runners now. Seems like a good time to test it!

Re: Ask HN: What CI/CD server/service do you use?

#30

Post by jazoom » Thu, Jan 26, 2017, 9:14 PM UTC

trcollinson wrote: post #29

jazoom wrote: post #28

Earlier quoted context omitted.

Thanks for taking the time to share your experience. What's the difference with your docker runner? I've been building docker images just fine without specifying a runner.

I run most of my runners as Docker runners and I was having issues with a Docker runner building a docker image. So I swapped to a different method. But I haven't tried it in a while. I might be able to use the same Docker runners now. Seems like a good time to test it!

Okay. Thanks again.