Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We cut our CI pipeline execution time in half

tinybird.co

Post reply on HN
Print view View original

11–20 of 100 posts

Re: We cut our CI pipeline execution time in half

#11

Post by packetlost »

danpalmer wrote: post #8
dijit wrote: post #2

I have a somewhat related question. I'm using gitlab-ci with it's docker executor, and overall I'm very happy with it. I use it on some rather beefy machines, but most of the CI time is not spent compiling, it is spent instead on setting up the environment. Are there any tips/tricks to speed up this startup time? I know stuff like ensuring that artifacts are not passed in if not needed can help a lot, but it seems th…

The short answer is "do as little as possible". What this means in practice is breaking down every step of CI, figuring out the dependencies for that step, and then ordering the graph of dependencies such that you start as much as possible as early as possible. This process also usually shows you where things are slow and what the critical path is. Unfortunately, doing this in most CI services is actually quite diffi…

It's so surprising to me that this is such a poorly supported paradigm in commodity CI systems. Caching artifacts and identifying slow stages is like... super important for scaling CI for large enough orgs. We need better tools!

Re: We cut our CI pipeline execution time in half

#12

Post by growse »

hinkley wrote: post #5
dijit wrote: post #2

I have a somewhat related question. I'm using gitlab-ci with it's docker executor, and overall I'm very happy with it. I use it on some rather beefy machines, but most of the CI time is not spent compiling, it is spent instead on setting up the environment. Are there any tips/tricks to speed up this startup time? I know stuff like ensuring that artifacts are not passed in if not needed can help a lot, but it seems th…

Pulling snapshots helps, particularly with slowdowns over time. Pulling deps is a problem that deserves its own initiatives. For me the controlling factor with build time and to a lesser extent production performance is to divorce visibility from vigilance. You can’t watch things 24/7 waiting to pounce on any little size or time regressions. You need to be able to audit periodically and narrow the problem to a commit…

> Pulling snapshots helps, particularly with slowdowns over time. Pulling deps is a problem that deserves its own initiatives.

I just had some success running android builds on a self-hosted github runner. One of the big setting up stages was having sdkamanger pull down large dependencies (SDK, emulator images etc.) on startup.

Forcing sdkmanager into http_only mode and pointing it at a properly-configured squid took a large percentage off the build time.

Similar story for the gradle build, where running a remote gradle cache node locally to the job means gradle steps get automatically cached without any magic CI pipeline steps.

Re: We cut our CI pipeline execution time in half

#13

Post by duped »

packetlost wrote: post #11
danpalmer wrote: post #8

Earlier quoted context omitted.

The short answer is "do as little as possible". What this means in practice is breaking down every step of CI, figuring out the dependencies for that step, and then ordering the graph of dependencies such that you start as much as possible as early as possible. This process also usually shows you where things are slow and what the critical path is. Unfortunately, doing this in most CI services is actually quite diffi…

It's so surprising to me that this is such a poorly supported paradigm in commodity CI systems. Caching artifacts and identifying slow stages is like... super important for scaling CI for large enough orgs. We need better tools!

The more time you spend debugging this and the worse job you do at it, the more money they make.

Re: We cut our CI pipeline execution time in half

#15

Post by nerdponx »

packetlost wrote: post #11
danpalmer wrote: post #8

Earlier quoted context omitted.

The short answer is "do as little as possible". What this means in practice is breaking down every step of CI, figuring out the dependencies for that step, and then ordering the graph of dependencies such that you start as much as possible as early as possible. This process also usually shows you where things are slow and what the critical path is. Unfortunately, doing this in most CI services is actually quite diffi…

It's so surprising to me that this is such a poorly supported paradigm in commodity CI systems. Caching artifacts and identifying slow stages is like... super important for scaling CI for large enough orgs. We need better tools!

Maybe it's because CI service providers don't want to be responsible for a lot cache storage.

Given how lacking this feature is, maybe a CI vendor could offer it as a premium paid feature.

Re: We cut our CI pipeline execution time in half

#16

Post by colechristensen »

dijit wrote: post #10
colechristensen wrote: post #4

Earlier quoted context omitted.

How long are we talking? Are the containers getting pulled from somewhere across the internet and it’s a network bottleneck?

it's a 30-90s 'setup' for a compile that usually lasts about 10-30s. The setup time is fairly constant even for very quick jobs. For longer jobs where it takes less of a percentage of the total time it's not a bother, like when we run integration tests for a few minutes.

Oh installing dependencies and things? Have prebaked images that already contain those things.

Re: We cut our CI pipeline execution time in half

#19

Post by hdjjhhvvhga »

dijit wrote: post #6
buttersbrian wrote: post #3

Earlier quoted context omitted.

How do you setup or provision your environment? And what does this environment look like?

Docker on Debian 11 bare metal with gitlab-ci installed the "blessed" way (by adding gitlabs apt repos). No optimisation to the baseOS other than mounting the /var/lib/docker on a RAID0 array with noatime on the volume and CPU mitigations disabled on the host Compilation is mostly go binaries (with the normal stuff like go vet/go test). Rarely it will do other things like commit-lint (javascript) or KICS/SNYK scannin…

Where do you keep your bare metal machines if I my ask? I wanted to do a similar setup a while ago (building/testing on Hetzner bare metal, deployments and the rest on AWS) but due to Amazon's pricing policy the cost of traffic would be enormous.

Re: We cut our CI pipeline execution time in half

#20

Post by dijit »

colechristensen wrote: post #16
dijit wrote: post #10

Earlier quoted context omitted.

it's a 30-90s 'setup' for a compile that usually lasts about 10-30s. The setup time is fairly constant even for very quick jobs. For longer jobs where it takes less of a percentage of the total time it's not a bother, like when we run integration tests for a few minutes.

Oh installing dependencies and things? Have prebaked images that already contain those things.

Yep, I'm talking about the time before my first command runs.
Post reply on HN