Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We cut our CI pipeline execution time in half

tinybird.co

Post reply on HN
Print view View original

81–90 of 100 posts

Re: We cut our CI pipeline execution time in half

#81

Post by hinkley »

growse wrote: post #12
hinkley wrote: post #5

Earlier quoted context omitted.

Pulling snapshots helps, particularly with slowdowns over time. Pulling deps is a problem that deserves its own initiatives. For me the controlling factor with build time and to a lesser extent production performance is to divorce visibility from vigilance. You can’t watch things 24/7 waiting to pounce on any little size or time regressions. You need to be able to audit periodically and narrow the problem to a commit…

> Pulling snapshots helps, particularly with slowdowns over time. Pulling deps is a problem that deserves its own initiatives. I just had some success running android builds on a self-hosted github runner. One of the big setting up stages was having sdkamanger pull down large dependencies (SDK, emulator images etc.) on startup. Forcing sdkmanager into http_only mode and pointing it at a properly-configured squid took…

dep caches are great until they aren't. We had to turn them off because we had some weird thing going on with 2 artifactory instances due to M&A issues and we were getting weird behaviors from it.

Being able to pull deltas sure is fast, but it also violates some of the principles of CI. Artifactory or similar tools can split the difference. As long as nobody is doing something dumb that is.

Re: We cut our CI pipeline execution time in half

#82

Post by tetha »

Scubabear68 wrote: post #39

Maybe I am just an old fuddy duddy conservative, but this struck me from the post: “In the grand scheme of things, one week isn’t that long. But to us, it felt like forever. We are constantly iterating and release multiple changes every day”. I assume they mean multiple production releases? Is this because the product lacks maturity or stability, or is it just your culture? I am asking because I am trying to imagine…

Mh, I'm interacting with teams with wildly different release strategies and stability requirements at work.

One of the more fundamental things actually pushing towards faster releases is what I call the relativistic deployment speed. We have products that will need at least 2 months to get a remotely deployable version ready. The average fast hotfix usually takes more like 4 months until an installation on a prod system actually can start. Our fastest products can go from code to prod in like 15 minutes with the automated tests being the bottleneck.

This in turn shapes choices for the product managers, but also for security. If something like Log4shell hit these slow products, I'd have to plan to be vulnerable for two months at least, and usually more like 4 - 8 months depending on the customers. I have no choice, because that's their light speed of deployment. No code goes to prod faster than two months latency. That, quite frankly, fucking sucks.

Other products were much better in that situation. We were lucky to have the right devs around, but we went from the decision to emergency log4shell at an utmost risky speed to the first log4shell patches in prod of many within 30 minutes.

However, that's not the normal speed, and that's when you get into the second decision area. Given a lightspeed of deployment, how fast do you want to go?

Some of our possibly faster moving products are B2B products, with a lot of internal training for support and consulting going into a release, and also training at customers happening for larger customers. This means, product chooses to only release bigger changes and heavily customer-visible changes every 6 weeks. They could do this a lot faster, but they choose to slow down because it fits their customers well. And for example, december is usually frozen entirely because customers want to.

But then there is the third decision area. What happens if there is an entirely customer invisible change, such as an optimization in database handling, some internal metric generation for an optimization, or an internal change to prepare a new feature for the next scheduled rollout? And we have the tested, vetted and working option to just push that into prod without downtime, with also gives us opportunity to build experience with, and confidence into our no-downtime deployment system? I don't see a reason why I wouldn't exercise this daily at least once.

Re: We cut our CI pipeline execution time in half

#83

Post by xboxnolifes »

teach wrote: post #38
berkle4455 wrote: post #18

Earlier quoted context omitted.

It’s literally not.

I'm tempted to just downvote you and move on with my life but I'm genuinely curious. Given that it's meaningless to Deploy something without Integrating the changes, what do you _actually_ mean by "You don’t need [CI]. Stick with CD only." Are you just talking about testing the changes? Help us out here.

I guess you could just have an unchanging project redeploy itself every hour or so.

Re: We cut our CI pipeline execution time in half

#84

Post by mariosisters »

chrisandchris wrote: post #67
mariosisters wrote: post #64

Earlier quoted context omitted.

I think it’s that you are an old fuddy duddy :P Actually, if you work with SMBs/enterprises, I agree with you on customer facing changes. In my past life we would ship very frequently (often more than once a day) but always had to feature flag changes that large clients might see or be affected by. Even something as simple as tweaking the layout of a core flow could cause support headaches and angry customers — custo…

I can only image how clean code looks & works that is full of feature flags. Glad that I don't need to do that to often :)

The right approach is to immediately remove the flags after rollout…

The actual approach is to maintain a million fucking feature flags, ensuring that almost all possible combinations are essentially untested… better hope you did a good job separating concerns!

Re: We cut our CI pipeline execution time in half

#85

Post by twic »

teach wrote: post #38
berkle4455 wrote: post #18

Earlier quoted context omitted.

It’s literally not.

I'm tempted to just downvote you and move on with my life but I'm genuinely curious. Given that it's meaningless to Deploy something without Integrating the changes, what do you _actually_ mean by "You don’t need [CI]. Stick with CD only." Are you just talking about testing the changes? Help us out here.

What if people push unmerged feature branches, and then the automation just deploys the most recent one?

I'm not advocating this, just to be clear.

Re: We cut our CI pipeline execution time in half

#86

Post by sdairs »

Scubabear68 wrote: post #69
sdairs wrote: post #59

Earlier quoted context omitted.

Hey, Al from Tinybird here (co-author of the post). We've made up to 20 production releases per day some days. It's transparent to our users, they aren't even aware the upgrade is happening, there's no upgrade button to hit, there's no downtime. We release often because we release small and fast. It's not like those 20 releases are always fundamentally changing the product. We would rather fix a minor bug or two and…

Thanks for the response Al! Very interesting. I agree different strokes for different folks, you guys seem to be on the extreme end of CI/CD. Have you done any sort of analysis you could share on what it costs to release up to 20 times per day?

We do track the real $ cost of time & materials, but tbh I don't think it's anything too exciting. I'll see what we can share!

Re: We cut our CI pipeline execution time in half

#87

Post by deely3 »

scott_w wrote: post #78
hedora wrote: post #53

Earlier quoted context omitted.

As a customer, if I find a competitor that does not do this, then I will switch to it. For example, I cancelled my netflix subscription because they are unable to reliably operate microservices, and the UI was always in some semi-broken state. As a software engineer, this stressed me out during my relaxing TV time. Even if continuous delivery is somehow reliably delivered, if the changes are customer visible, then th…

And yet the numbers show that companies that continuously delivery updates massively outperform those that don’t. You claim to make decisions based on a company’s engineering practices but I can guarantee that you have no idea about the engineering practices of any companies that you do support actually are.

Wait, can I see these numbers grouped by industries if possible? Please?

Re: We cut our CI pipeline execution time in half

#88

Post by scott_w »

deely3 wrote: post #87
scott_w wrote: post #78

Earlier quoted context omitted.

And yet the numbers show that companies that continuously delivery updates massively outperform those that don’t. You claim to make decisions based on a company’s engineering practices but I can guarantee that you have no idea about the engineering practices of any companies that you do support actually are.

Wait, can I see these numbers grouped by industries if possible? Please?

Read any book on the subject, they’ve done the research for you. The Phoenix Project, The State of DevOps Report.

Spend some time in companies that move slow vs fast and you’ll see the difference in their success first hand. You’ll see the metrics on their incidents and severity and customer satisfaction with them.

Oh, and the fact two companies mentioned (Google and Facebook) are two of the most successful companies on earth.

Re: We cut our CI pipeline execution time in half

#89

Post by igetspam »

Scubabear68 wrote: post #39

Maybe I am just an old fuddy duddy conservative, but this struck me from the post: “In the grand scheme of things, one week isn’t that long. But to us, it felt like forever. We are constantly iterating and release multiple changes every day”. I assume they mean multiple production releases? Is this because the product lacks maturity or stability, or is it just your culture? I am asking because I am trying to imagine…

Read the State of DevOps reports over the years and you'll see why this is the direction we're all heading now. It turns out all that safety we thought we were building by making complex commit flows to multiple branches and environments was not only more complex than it needed to be but has also slowed us down and not made things better. Truly based development is back again and this time with data. Push early, push often, push small changes and iterate quickly. It's not just easier but it also seems to increase quality. (There are a lot of reasons why this turns out to be true. Read Accelerate. I won't do a better job explaining in a comment.)

Re: We cut our CI pipeline execution time in half

#90

Post by igetspam »

deely3 wrote: post #87
scott_w wrote: post #78

Earlier quoted context omitted.

And yet the numbers show that companies that continuously delivery updates massively outperform those that don’t. You claim to make decisions based on a company’s engineering practices but I can guarantee that you have no idea about the engineering practices of any companies that you do support actually are.

Wait, can I see these numbers grouped by industries if possible? Please?

Yes. Read Accelerate and read the State of DevOps reports from 2017 through 2022. The reports have the data and explain their methodologies for evaluating said data. It's all there for you to consume.
Post reply on HN