Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We cut our CI pipeline execution time in half

tinybird.co

Post reply on HN
Print view View original

51–60 of 100 posts

Re: We cut our CI pipeline execution time in half

#51

Post by kvdveer »

actionfromafar wrote: post #41
dijit wrote: post #33

Earlier quoted context omitted.

Main reason is because honestly I'm too scared of a dirty filesystem wrecking builds.

So, clean it, run in a tmpfs or whatever.

Docker images in CI are typically just that: a tmpfs with a chroot and some network isolation. If you have it working once, youre pretty much guaranteed it will work again.

Doing this on bare metal with a script to clean the FS, ensure correct dependencies and maybe isolate the network (for safe unit tests), means you're just reimplementing much a non-trivial portion of docker or other container tools. Maybe that's worth it, but without justification, it just smells like risky NIH to me.

Re: We cut our CI pipeline execution time in half

#52

Post by danpalmer »

nerdponx wrote: post #15
packetlost wrote: post #11

Earlier quoted context omitted.

It's so surprising to me that this is such a poorly supported paradigm in commodity CI systems. Caching artifacts and identifying slow stages is like... super important for scaling CI for large enough orgs. We need better tools!

Maybe it's because CI service providers don't want to be responsible for a lot cache storage. Given how lacking this feature is, maybe a CI vendor could offer it as a premium paid feature.

CircleCI sort of do! They have something called Docker Layer Caching, which basically puts all the Docker layers from your previous build on the execution machine.

The problem is that it's a) very slow to download those layers from their cache storage, and b) very expensive. It works out to costing ~20 minutes of build time.

Re: We cut our CI pipeline execution time in half

#53

Post by hedora »

scott_w wrote: post #44
Scubabear68 wrote: post #39

Maybe I am just an old fuddy duddy conservative, but this struck me from the post: “In the grand scheme of things, one week isn’t that long. But to us, it felt like forever. We are constantly iterating and release multiple changes every day”. I assume they mean multiple production releases? Is this because the product lacks maturity or stability, or is it just your culture? I am asking because I am trying to imagine…

This is considered the norm for high performing product teams in the modern day. We keep customers happy because we push changes live incrementally, reduce our chances of major outages and improve our response time when they do occur.

As a customer, if I find a competitor that does not do this, then I will switch to it.

For example, I cancelled my netflix subscription because they are unable to reliably operate microservices, and the UI was always in some semi-broken state. As a software engineer, this stressed me out during my relaxing TV time.

Even if continuous delivery is somehow reliably delivered, if the changes are customer visible, then they break my muscle memory, and increase my cognitive load -- I have to re-learn the damned UI every fucking time I log in. If the changes are not customer visible, then what business value to they deliver?

Re: We cut our CI pipeline execution time in half

#54

Post by Scubabear68 »

recfab wrote: post #45
Scubabear68 wrote: post #39

Maybe I am just an old fuddy duddy conservative, but this struck me from the post: “In the grand scheme of things, one week isn’t that long. But to us, it felt like forever. We are constantly iterating and release multiple changes every day”. I assume they mean multiple production releases? Is this because the product lacks maturity or stability, or is it just your culture? I am asking because I am trying to imagine…

This just sounds like Continuous Delivery. We never achieved it in my last job, so I can't speak from experience, but my understanding is that typically "deploy" is separated from "release" using feature flags of some kind.

The article starts with “Last year, we made the difficult decision to stop deploying any changes to production for one week” and goes on to talk about releases.

In that context I assume this means they make multiple production releases per day (which makes me shudder). I am curious how they do this while maintaining high quality and not driving customers insane.

Re: We cut our CI pipeline execution time in half

#55

Post by fragmede »

hedora wrote: post #49
brightball wrote: post #34

Earlier quoted context omitted.

If you have it, it’s awesome. You can get parallel execution of so much, spin up environments for each branch for QA and dynamic scans. IMO it’s the optimal use case for K8s

You have to be at a certain scale for k8s to make sense in a CI environment. In particular, it needs to be economical to spend 10-50% of a full time employee to maintain the Kubernetes cluster (even if it is some managed thing like EKS). Also, the duty cycle on the 21 nodes needs to be low enough to justify the complexity over just buying 21 computers (or getting annual pricing on 21 VMs). You could use spot instance…

If you've only got 21 worker machines that probably works out, but if you've got 210, or 2,100 of them to spin up/down, I'd rather be dealing with yaml config (even though I hate yaml config) rather than get PXE booting working for an on-prem cluster.

Re: We cut our CI pipeline execution time in half

#56

Post by hedora »

dijit wrote: post #2

I have a somewhat related question. I'm using gitlab-ci with it's docker executor, and overall I'm very happy with it. I use it on some rather beefy machines, but most of the CI time is not spent compiling, it is spent instead on setting up the environment. Are there any tips/tricks to speed up this startup time? I know stuff like ensuring that artifacts are not passed in if not needed can help a lot, but it seems th…

Make sure your docker build is being cached properly, and break infrequently running stuff into their own steps, then move them to the top of the docker file.

Crucially: Make sure that the large layers say they are "cached" when you rebuild the container. Docker goes out of its way to make this difficult in CI environments. The fact that it works on your laptop doesn't mean that it will be able to cache the big layers in CI.

Once you've done that, make sure that the CI machines are actually pulling the big layers from their local docker cache.

30-90 seconds to pull docker images for each run of a golang project's CI environment is too high. You might look into using "go mod vendor" to download the dependencies early in the docker build, then using a symlink and "--mod=vendor" to tell the tests to use an out-of-tree vendor directory. (I haven't tested this; presumably go will follow symlinks...)

Re: We cut our CI pipeline execution time in half

#57

Post by fragmede »

Scubabear68 wrote: post #54
recfab wrote: post #45

Earlier quoted context omitted.

This just sounds like Continuous Delivery. We never achieved it in my last job, so I can't speak from experience, but my understanding is that typically "deploy" is separated from "release" using feature flags of some kind.

The article starts with “Last year, we made the difficult decision to stop deploying any changes to production for one week” and goes on to talk about releases. In that context I assume this means they make multiple production releases per day (which makes me shudder). I am curious how they do this while maintaining high quality and not driving customers insane.

> multiple production releases per day

Most (but not all) SaaS businesses are expected to these days, so I'm curious what your business/industry is that not only that you don't, but that it gives you the shudders.

Re: We cut our CI pipeline execution time in half

#58

Post by Scubabear68 »

fragmede wrote: post #57
Scubabear68 wrote: post #54

Earlier quoted context omitted.

The article starts with “Last year, we made the difficult decision to stop deploying any changes to production for one week” and goes on to talk about releases. In that context I assume this means they make multiple production releases per day (which makes me shudder). I am curious how they do this while maintaining high quality and not driving customers insane.

> multiple production releases per day Most (but not all) SaaS businesses are expected to these days, so I'm curious what your business/industry is that not only that you don't, but that it gives you the shudders.

I consult across a few different industries, but it includes SaaS offerings. Many places could do multiple deploys to prod a day, but choose weekly releases or other longer cadences. This is to allow for documentation, client notifications, etc. It also is more efficient, constantly releasing requires a lot of resources, as this blog implies they are churning dozens of K8 pods several times a day.

I would think constant releasing would also make debugging prod issues pure hell.

Re: We cut our CI pipeline execution time in half

#59

Post by sdairs »

Scubabear68 wrote: post #54
recfab wrote: post #45

Earlier quoted context omitted.

This just sounds like Continuous Delivery. We never achieved it in my last job, so I can't speak from experience, but my understanding is that typically "deploy" is separated from "release" using feature flags of some kind.

The article starts with “Last year, we made the difficult decision to stop deploying any changes to production for one week” and goes on to talk about releases. In that context I assume this means they make multiple production releases per day (which makes me shudder). I am curious how they do this while maintaining high quality and not driving customers insane.

Hey, Al from Tinybird here (co-author of the post). We've made up to 20 production releases per day some days. It's transparent to our users, they aren't even aware the upgrade is happening, there's no upgrade button to hit, there's no downtime. We release often because we release small and fast. It's not like those 20 releases are always fundamentally changing the product. We would rather fix a minor bug or two and get that out to our customers ASAP, than hold on to it for a few months and drop a huge change. In a vast majority of cases, a user won't even consciously notice something changed.

Doing this kind of fast iteration has its risks, but it has its benefits too. We de-risk it, in part, by having extensive CI, which is why it was so important to us that the CI is fast & reliable.

Delivering larger, less-frequent updates has its own risks. You're not practising your release process as frequently, so it's a much bigger event. You're pushing many, many more changes in one go, so there's a lot more surface area for something to go wrong, and rolling it back is a much bigger job. And dropping many/bigger changes to the user experience is much more noticeable.

Again, this isn't the right process for everyone, but it works for us and its how we've managed to build a product that delivers value to our users.

Re: We cut our CI pipeline execution time in half

#60

Post by brightball »

danpalmer wrote: post #50
brightball wrote: post #36

Earlier quoted context omitted.

I have found Gitlab and runners the best option here.

The problem I had with GitLab was that the mechanisms for controlling dependencies between stages were fairly basic. They only added them in ~2020 I think, and they weren't well documented. Additionally, there's no cache guarantees between jobs within one execution. This means that you can't reliably cache an artifact in one job, and then share it with multiple downstream jobs. It mostly works, but it's hard to debug…

Dependency controls have improved quite a bit. They went through a couple of variations of this and the current solution is nice.

I haven't ever run into an issue with artifact hand off to this point though. Maybe it's one of the more rare concerns, but it's not something I've experienced (fortunately). I imagine it would be a concern to debug though.

Post reply on HN