Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We cut our CI pipeline execution time in half

tinybird.co

Post reply on HN
Print view View original

71–80 of 100 posts

Re: We cut our CI pipeline execution time in half

#71

Post by alrocar »

imiric wrote: post #70

> We noticed a strong correlation between crazy utilization spikes and CI failure rates. This is interesting, and is something I've also suspected on many CI systems that offer free public runners (CircleCI, GitHub Actions, etc.). For seemingly no reason at all, tests were very flaky and unstable in CI, which couldn't be reproduced on local machines. I tried everything from resource-limited containers, to identically…

Yep, default runners in most CI platforms shared resources so they are prone to produce flakiness (depending on your set up).

That was one of the reasons we ended up setting up our own runners. Didn't mention in the post but we use spot VM instances.

Re: We cut our CI pipeline execution time in half

#72

Post by drewcoo »

recfab wrote: post #45
Scubabear68 wrote: post #39

Maybe I am just an old fuddy duddy conservative, but this struck me from the post: “In the grand scheme of things, one week isn’t that long. But to us, it felt like forever. We are constantly iterating and release multiple changes every day”. I assume they mean multiple production releases? Is this because the product lacks maturity or stability, or is it just your culture? I am asking because I am trying to imagine…

This just sounds like Continuous Delivery. We never achieved it in my last job, so I can't speak from experience, but my understanding is that typically "deploy" is separated from "release" using feature flags of some kind.

It sounds like continuous deployment, not continuous delivery.

Continuous deployment deploys code to production frequently, as soon as it's ready.

Continuous delivery has some ready-to deliver branch that's constantly being updated as above, but they're not deployed to production until someone (Product Owner?) or something (Yay - end of sprint!) triggers it.

Different people may use the word release for at least this many things: 1) a deployment, 2) an unveiling via feature flags, 3) a public announcement despite the code already having been live.

Re: We cut our CI pipeline execution time in half

#73

Post by alrocar »

Scubabear68 wrote: post #39

Maybe I am just an old fuddy duddy conservative, but this struck me from the post: “In the grand scheme of things, one week isn’t that long. But to us, it felt like forever. We are constantly iterating and release multiple changes every day”. I assume they mean multiple production releases? Is this because the product lacks maturity or stability, or is it just your culture? I am asking because I am trying to imagine…

Yep, it's part of our culture, we do many releases per day to constantly iterate things. Also as in other projects there are maintenance and bug fixing we want to bring to production as soon as possible.

Our context is the one of a startup that is constantly validating things, also in our context a release does not necessarily mean releasing to the users, sometimes stuff is behind feature flags or for beta testing.

Re: We cut our CI pipeline execution time in half

#75

Post by alrocar »

teach wrote: post #31

Am I a curmudgeon? Not to take away from this cool writeup, but I'm familiar with a few CI/CD tools, particularly QuickBuild, Jenkins and Spinnaker. So this jumped out at me: > Our CI process was pretty standard: Every commit in an MR triggered a GitLab Pipeline, which consisted of several jobs. me: nodding silently > Those jobs would run in an auto-scaling Kubernetes cluster with up to 21 nodes me: what the actual d…

Thanks for this comment. I guess there's sometimes we (developers) take things for granted when they are not, and that puts a lot of pressure on us instead of celebrating our wins.

I would change now "pretty standard" by "we don't invented the wheel" xD :pray:, in the end I wanted to mean we use existing tools and "just" put them together

Re: We cut our CI pipeline execution time in half

#76

Post by alrocar »

berkle4455 wrote: post #14

CI has been such a productivity killer. You don’t need it. Stick with CD only and you can ship.

I'm really interested in different points of view.

I guess you mean the kind of trunk based development? But still some sort of CI happens, maybe locally.

Never worked in a different way than using a local / remote CI pipeline, that's why I'm curious.

Re: We cut our CI pipeline execution time in half

#77

Post by drewcoo »

Scubabear68 wrote: post #39

Maybe I am just an old fuddy duddy conservative, but this struck me from the post: “In the grand scheme of things, one week isn’t that long. But to us, it felt like forever. We are constantly iterating and release multiple changes every day”. I assume they mean multiple production releases? Is this because the product lacks maturity or stability, or is it just your culture? I am asking because I am trying to imagine…

Hey there! I'm an old fuddy duddy, too.

Continuous deployment has been around long enough that even IBM (remember never getting fired for buying IBM?) talks about it.

https://www.ibm.com/topics/continuous-deployment

"Dark deploys" and "feature flags" are often used to keep customers safe from incomplete features while still giving all of the advantages of CD plus allowing testing in production.

I'd never heard of Flagship, but this is a nice writeup on that (kudos, Flagship.io):

https://www.flagship.io/glossary/dark-launch/

Re: We cut our CI pipeline execution time in half

#78

Post by scott_w »

hedora wrote: post #53
scott_w wrote: post #44

Earlier quoted context omitted.

This is considered the norm for high performing product teams in the modern day. We keep customers happy because we push changes live incrementally, reduce our chances of major outages and improve our response time when they do occur.

As a customer, if I find a competitor that does not do this, then I will switch to it. For example, I cancelled my netflix subscription because they are unable to reliably operate microservices, and the UI was always in some semi-broken state. As a software engineer, this stressed me out during my relaxing TV time. Even if continuous delivery is somehow reliably delivered, if the changes are customer visible, then th…

And yet the numbers show that companies that continuously delivery updates massively outperform those that don’t. You claim to make decisions based on a company’s engineering practices but I can guarantee that you have no idea about the engineering practices of any companies that you do support actually are.

Re: We cut our CI pipeline execution time in half

#79

Post by tempest_ »

chrisandchris wrote: post #67
mariosisters wrote: post #64

Earlier quoted context omitted.

I think it’s that you are an old fuddy duddy :P Actually, if you work with SMBs/enterprises, I agree with you on customer facing changes. In my past life we would ship very frequently (often more than once a day) but always had to feature flag changes that large clients might see or be affected by. Even something as simple as tweaking the layout of a core flow could cause support headaches and angry customers — custo…

I can only image how clean code looks & works that is full of feature flags. Glad that I don't need to do that to often :)

They keep adding them until they need an internal library to manage their collection of feature flags.

Then after a while they graduate to feature flags as a service (of which there are a bunch of cloud services trying to have a go at)

Re: We cut our CI pipeline execution time in half

#80

Post by sdairs »

chrisandchris wrote: post #68
sdairs wrote: post #59

Earlier quoted context omitted.

Hey, Al from Tinybird here (co-author of the post). We've made up to 20 production releases per day some days. It's transparent to our users, they aren't even aware the upgrade is happening, there's no upgrade button to hit, there's no downtime. We release often because we release small and fast. It's not like those 20 releases are always fundamentally changing the product. We would rather fix a minor bug or two and…

It's interesting how you differ between "deploy ASAP, within half a day" and "keep the fix a few months back". Like, is there nothing in between? Like once a week, once every other week? If you would have to estimate, wouldn't be there less bugs if you would deploy less fast (and use this time for validation)?

Sure :D

I've worked on products across the spectrum, Enterprise software that releases twice a year, smaller stuff that ships once a month, and now a SaaS that releases many times a day. I don't have the data to compare, but I genuinely don't believe the rate of bugs was materially different between any of them.

We make faster changes, but the changes are much smaller, and so the surface area to test is smaller. The more changes you make, the more time you need to validate. If you make a weeks worth of changes, you're going to need an appropriately longer validation cycle than if you make 1 change. It scales, and in my experience, many products with slower release cycles aren't appropriately scaling up the validation time to match.

That said, this doesn't necessarily mean we're writing code and 30 minutes later its in production...there's still an iterative dev cycle with lots of validation happening...but if something is ready to go, ship it!

Post reply on HN