We cut our CI pipeline execution time in half
1–10 of 100 posts
Re: We cut our CI pipeline execution time in half
#2I have a somewhat related question.
I'm using gitlab-ci with it's docker executor, and overall I'm very happy with it.
I use it on some rather beefy machines, but most of the CI time is not spent compiling, it is spent instead on setting up the environment.
Are there any tips/tricks to speed up this startup time? I know stuff like ensuring that artifacts are not passed in if not needed can help a lot, but it seems that most of the execution time is simply spent waiting for docker to spin up a container.
Re: We cut our CI pipeline execution time in half
#3I have a somewhat related question. I'm using gitlab-ci with it's docker executor, and overall I'm very happy with it. I use it on some rather beefy machines, but most of the CI time is not spent compiling, it is spent instead on setting up the environment. Are there any tips/tricks to speed up this startup time? I know stuff like ensuring that artifacts are not passed in if not needed can help a lot, but it seems th…
How do you setup or provision your environment? And what does this environment look like?
Re: We cut our CI pipeline execution time in half
#4I have a somewhat related question. I'm using gitlab-ci with it's docker executor, and overall I'm very happy with it. I use it on some rather beefy machines, but most of the CI time is not spent compiling, it is spent instead on setting up the environment. Are there any tips/tricks to speed up this startup time? I know stuff like ensuring that artifacts are not passed in if not needed can help a lot, but it seems th…
How long are we talking? Are the containers getting pulled from somewhere across the internet and it’s a network bottleneck?
Re: We cut our CI pipeline execution time in half
#5I have a somewhat related question. I'm using gitlab-ci with it's docker executor, and overall I'm very happy with it. I use it on some rather beefy machines, but most of the CI time is not spent compiling, it is spent instead on setting up the environment. Are there any tips/tricks to speed up this startup time? I know stuff like ensuring that artifacts are not passed in if not needed can help a lot, but it seems th…
Pulling snapshots helps, particularly with slowdowns over time. Pulling deps is a problem that deserves its own initiatives.
For me the controlling factor with build time and to a lesser extent production performance is to divorce visibility from vigilance. You can’t watch things 24/7 waiting to pounce on any little size or time regressions. You need to be able to audit periodically and narrow the problem to a commit or at least an hour in a day when the problem happened. Otherwise nobody will be bothered to look and it’s just a tragedy of the commons.
Graphs work well. Build time, test count, slow test count, artifact sizes, and so on.
Re: We cut our CI pipeline execution time in half
#6I have a somewhat related question. I'm using gitlab-ci with it's docker executor, and overall I'm very happy with it. I use it on some rather beefy machines, but most of the CI time is not spent compiling, it is spent instead on setting up the environment. Are there any tips/tricks to speed up this startup time? I know stuff like ensuring that artifacts are not passed in if not needed can help a lot, but it seems th…
How do you setup or provision your environment? And what does this environment look like?
Docker on Debian 11 bare metal with gitlab-ci installed the "blessed" way (by adding gitlabs apt repos).
No optimisation to the baseOS other than mounting the /var/lib/docker on a RAID0 array with noatime on the volume and CPU mitigations disabled on the host
Compilation is mostly go binaries (with the normal stuff like go vet/go test).
Rarely it will do other things like commit-lint (javascript) or KICS/SNYK scanning.
the machines themselves are Dual EPYC 7313 w/ 256G DDR4.
Re: We cut our CI pipeline execution time in half
#7I have a somewhat related question. I'm using gitlab-ci with it's docker executor, and overall I'm very happy with it. I use it on some rather beefy machines, but most of the CI time is not spent compiling, it is spent instead on setting up the environment. Are there any tips/tricks to speed up this startup time? I know stuff like ensuring that artifacts are not passed in if not needed can help a lot, but it seems th…
Set up the environment on the runner, not in the job: https://docs.gitlab.com/ee/ci/runners/configure_runners.html...
At the very least, see if you can keep heavy dependencies on the local network rather than depending on the internet.
Re: We cut our CI pipeline execution time in half
#8I have a somewhat related question. I'm using gitlab-ci with it's docker executor, and overall I'm very happy with it. I use it on some rather beefy machines, but most of the CI time is not spent compiling, it is spent instead on setting up the environment. Are there any tips/tricks to speed up this startup time? I know stuff like ensuring that artifacts are not passed in if not needed can help a lot, but it seems th…
The short answer is "do as little as possible". What this means in practice is breaking down every step of CI, figuring out the dependencies for that step, and then ordering the graph of dependencies such that you start as much as possible as early as possible. This process also usually shows you where things are slow and what the critical path is.
Unfortunately, doing this in most CI services is actually quite difficult. It usually means a complex graph of execution, complex cache usage, and being careful to not re-generate artifacts you don't need to.
In my experience, building this, at a level of reliability necessary for a team of more than a few devs, is hard. Jenkins can do it reliably, but doing it fast is hard because the caching primitives are poor. Circle and Gitlab can do it quickly, but the execution dependency primitives aren't great and the caches can be unreliable. Circle also has _terrible_ network speeds, so doing too much caching slows down builds. GitHub Actions is pretty good for all of this, but it's still a ton of work.
The best answer is to use a build system or CI system that is modelled in a better way. Things like Bazel essentially manage this graph for you in a very smart way, but they only really work when you have a CI system designed to run Bazel jobs, and there aren't many of these that I've seen. It's a huge paradigm shift, and requires quite a lot of dev work to make happen.
Re: We cut our CI pipeline execution time in half
#9I have a somewhat related question. I'm using gitlab-ci with it's docker executor, and overall I'm very happy with it. I use it on some rather beefy machines, but most of the CI time is not spent compiling, it is spent instead on setting up the environment. Are there any tips/tricks to speed up this startup time? I know stuff like ensuring that artifacts are not passed in if not needed can help a lot, but it seems th…
How long are we talking? Are the containers getting pulled from somewhere across the internet and it’s a network bottleneck?
This is what I’m working on next week. The majority of time is spent building the first n numbers of our Dockerfiles (which aren’t cached in our test/deploy pipeline).
I’ll be baking some images with dependencies included, so the only stuff in the updated Dockerfile will be pulling the pre baked images from our registry and commands to build and run our app code.
Re: We cut our CI pipeline execution time in half
#10I have a somewhat related question. I'm using gitlab-ci with it's docker executor, and overall I'm very happy with it. I use it on some rather beefy machines, but most of the CI time is not spent compiling, it is spent instead on setting up the environment. Are there any tips/tricks to speed up this startup time? I know stuff like ensuring that artifacts are not passed in if not needed can help a lot, but it seems th…
How long are we talking? Are the containers getting pulled from somewhere across the internet and it’s a network bottleneck?
it's a 30-90s 'setup' for a compile that usually lasts about 10-30s.
The setup time is fairly constant even for very quick jobs.
For longer jobs where it takes less of a percentage of the total time it's not a bother, like when we run integration tests for a few minutes.