Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We cut our CI pipeline execution time in half

tinybird.co

Post reply on HN
Print view View original

61–70 of 100 posts

Re: We cut our CI pipeline execution time in half

#61

Post by brightball »

hedora wrote: post #49
brightball wrote: post #34

Earlier quoted context omitted.

If you have it, it’s awesome. You can get parallel execution of so much, spin up environments for each branch for QA and dynamic scans. IMO it’s the optimal use case for K8s

You have to be at a certain scale for k8s to make sense in a CI environment. In particular, it needs to be economical to spend 10-50% of a full time employee to maintain the Kubernetes cluster (even if it is some managed thing like EKS). Also, the duty cycle on the 21 nodes needs to be low enough to justify the complexity over just buying 21 computers (or getting annual pricing on 21 VMs). You could use spot instance…

I actually find that K8s in a CI environment is a better use case than in production environments.

In production, you're going to have clearly defined deployment rules, traffic patterns, scalability approaches where the code for each service probably belongs on it's own VM rather than sharing cluster resources.

In non-production environment, you can feel much more free to overload what's deployed on a node because it's not seeing production traffic. You could have a single k8s instance with 30 different environments (each with their own web, worker, databases, redis, etc) for 30 different branches that represent the issues moving through the pipeline from 5 developers. In prod, ever piece of that would be better represented by it's own VM.

Re: We cut our CI pipeline execution time in half

#62

Post by charrondev »

dijit wrote: post #2

I have a somewhat related question. I'm using gitlab-ci with it's docker executor, and overall I'm very happy with it. I use it on some rather beefy machines, but most of the CI time is not spent compiling, it is spent instead on setting up the environment. Are there any tips/tricks to speed up this startup time? I know stuff like ensuring that artifacts are not passed in if not needed can help a lot, but it seems th…

I had similar problems with CircleCI and it’s docker executor. We recently switched to GitHub actions and the following led to huge improvements:

- much faster network speeds. - We no longer run on the docker executor. Instead we run on Ubuntu. These boot in a second or 2 pretty consistently. - the bulk of our test suite was able to be pulled out of docker entirely (a lot of jest, and PHPUnit tests). - we have a bigger suite of E2E PHPUnit tests that we spin up a whole docker compose stack for. These are slower but still manageable.

Parallelism is key in all of this too. Our backend test suite has a full execution time of something like 250 minutes, but we just split it over a bunch of small workers and the whole thing completes in about 8 minutes.

Re: We cut our CI pipeline execution time in half

#63

Post by kyriakos »

dijit wrote: post #2

I have a somewhat related question. I'm using gitlab-ci with it's docker executor, and overall I'm very happy with it. I use it on some rather beefy machines, but most of the CI time is not spent compiling, it is spent instead on setting up the environment. Are there any tips/tricks to speed up this startup time? I know stuff like ensuring that artifacts are not passed in if not needed can help a lot, but it seems th…

I noticed that many times using cache makes gitlab ci take longer than just fetching node dependencies again via npm install.

Re: We cut our CI pipeline execution time in half

#64

Post by mariosisters »

Scubabear68 wrote: post #39

Maybe I am just an old fuddy duddy conservative, but this struck me from the post: “In the grand scheme of things, one week isn’t that long. But to us, it felt like forever. We are constantly iterating and release multiple changes every day”. I assume they mean multiple production releases? Is this because the product lacks maturity or stability, or is it just your culture? I am asking because I am trying to imagine…

I think it’s that you are an old fuddy duddy :P

Actually, if you work with SMBs/enterprises, I agree with you on customer facing changes. In my past life we would ship very frequently (often more than once a day) but always had to feature flag changes that large clients might see or be affected by. Even something as simple as tweaking the layout of a core flow could cause support headaches and angry customers — customers worth 10s of thousands of dollars per month. Is it worth losing a customer to CD a new button placement?

Re: We cut our CI pipeline execution time in half

#65

Post by mariosisters »

lelandfe wrote: post #46
Scubabear68 wrote: post #39

Maybe I am just an old fuddy duddy conservative, but this struck me from the post: “In the grand scheme of things, one week isn’t that long. But to us, it felt like forever. We are constantly iterating and release multiple changes every day”. I assume they mean multiple production releases? Is this because the product lacks maturity or stability, or is it just your culture? I am asking because I am trying to imagine…

That’s continuous delivery, right? You make great tests and you should feel comfortable releasing after review.

In my experience and to parent’s point, it’s not about your comfort it’s about documenting, notifying clients, updating support, etc. All the non-code parts of selling software. As you suggest if the code has been reviewed, tested and merged, it “should” be ready to go. Right?

Re: We cut our CI pipeline execution time in half

#66

Post by sleepychu »

duped wrote: post #13
packetlost wrote: post #11

Earlier quoted context omitted.

It's so surprising to me that this is such a poorly supported paradigm in commodity CI systems. Caching artifacts and identifying slow stages is like... super important for scaling CI for large enough orgs. We need better tools!

The more time you spend debugging this and the worse job you do at it, the more money they make.

That feels counterintuitive to me. I would probably use even more CI minutes if they had higher value.

Re: We cut our CI pipeline execution time in half

#67

Post by chrisandchris »

mariosisters wrote: post #64
Scubabear68 wrote: post #39

Maybe I am just an old fuddy duddy conservative, but this struck me from the post: “In the grand scheme of things, one week isn’t that long. But to us, it felt like forever. We are constantly iterating and release multiple changes every day”. I assume they mean multiple production releases? Is this because the product lacks maturity or stability, or is it just your culture? I am asking because I am trying to imagine…

I think it’s that you are an old fuddy duddy :P Actually, if you work with SMBs/enterprises, I agree with you on customer facing changes. In my past life we would ship very frequently (often more than once a day) but always had to feature flag changes that large clients might see or be affected by. Even something as simple as tweaking the layout of a core flow could cause support headaches and angry customers — custo…

I can only image how clean code looks & works that is full of feature flags. Glad that I don't need to do that to often :)

Re: We cut our CI pipeline execution time in half

#68

Post by chrisandchris »

sdairs wrote: post #59
Scubabear68 wrote: post #54

Earlier quoted context omitted.

The article starts with “Last year, we made the difficult decision to stop deploying any changes to production for one week” and goes on to talk about releases. In that context I assume this means they make multiple production releases per day (which makes me shudder). I am curious how they do this while maintaining high quality and not driving customers insane.

Hey, Al from Tinybird here (co-author of the post). We've made up to 20 production releases per day some days. It's transparent to our users, they aren't even aware the upgrade is happening, there's no upgrade button to hit, there's no downtime. We release often because we release small and fast. It's not like those 20 releases are always fundamentally changing the product. We would rather fix a minor bug or two and…

It's interesting how you differ between "deploy ASAP, within half a day" and "keep the fix a few months back".

Like, is there nothing in between? Like once a week, once every other week?

If you would have to estimate, wouldn't be there less bugs if you would deploy less fast (and use this time for validation)?

Re: We cut our CI pipeline execution time in half

#69

Post by Scubabear68 »

sdairs wrote: post #59
Scubabear68 wrote: post #54

Earlier quoted context omitted.

The article starts with “Last year, we made the difficult decision to stop deploying any changes to production for one week” and goes on to talk about releases. In that context I assume this means they make multiple production releases per day (which makes me shudder). I am curious how they do this while maintaining high quality and not driving customers insane.

Hey, Al from Tinybird here (co-author of the post). We've made up to 20 production releases per day some days. It's transparent to our users, they aren't even aware the upgrade is happening, there's no upgrade button to hit, there's no downtime. We release often because we release small and fast. It's not like those 20 releases are always fundamentally changing the product. We would rather fix a minor bug or two and…

Thanks for the response Al!

Very interesting. I agree different strokes for different folks, you guys seem to be on the extreme end of CI/CD.

Have you done any sort of analysis you could share on what it costs to release up to 20 times per day?

Re: We cut our CI pipeline execution time in half

#70

Post by imiric »

> We noticed a strong correlation between crazy utilization spikes and CI failure rates.

This is interesting, and is something I've also suspected on many CI systems that offer free public runners (CircleCI, GitHub Actions, etc.).

For seemingly no reason at all, tests were very flaky and unstable in CI, which couldn't be reproduced on local machines. I tried everything from resource-limited containers, to identically spec'd VMs, and never was able to reproduce certain failures. This made issues very hard to troubleshoot and fix.

Of course, you might say that this unstable environment surfaced race conditions in our tests or product, and that's true, but it's incredibly frustrating to have random failures that are impossible to reproduce locally, and having to wait for the long experiment-push-wait for CI development loop.

I suspect this is caused by over provisioning of the underlying hardware, where many VMs are competing for the same resources. This seems quite frequent on Azure (GH Actions).

In the article's case they patched it by making their environment more stable, which is a solution we can't do on public runners, but I'd caution them that they're only patching the issue, and not really fixing the root cause. The flakiness still exists in their code, and is just not visible when the system is not under stress, but will surface again when you least want it to, possibly in production.

Post reply on HN