41–50 of 100 posts
Am I a curmudgeon? Not to take away from this cool writeup, but I'm familiar with a few CI/CD tools, particularly QuickBuild, Jenkins and Spinnaker. So this jumped out at me: > Our CI process was pretty standard: Every commit in an MR triggered a GitLab Pipeline, which consisted of several jobs. me: nodding silently > Those jobs would run in an auto-scaling Kubernetes cluster with up to 21 nodes me: what the actual d…
Gitlab makes it pretty easy to just toss a ci runner process on a vm or a physical box. You can get real far with a couple rack servers and some xeons for < $1000. You do have to over provision if your work load is not very consistent ( and of course pay for the power and rack space, and someone to mind them from time to time).
Am I a curmudgeon? Not to take away from this cool writeup, but I'm familiar with a few CI/CD tools, particularly QuickBuild, Jenkins and Spinnaker. So this jumped out at me: > Our CI process was pretty standard: Every commit in an MR triggered a GitLab Pipeline, which consisted of several jobs. me: nodding silently > Those jobs would run in an auto-scaling Kubernetes cluster with up to 21 nodes me: what the actual d…
Even if you are using SaaS GitLab, there are still good reasons to have custom runners, and kube is one option for running them.
Maybe I am just an old fuddy duddy conservative, but this struck me from the post: “In the grand scheme of things, one week isn’t that long. But to us, it felt like forever. We are constantly iterating and release multiple changes every day”. I assume they mean multiple production releases? Is this because the product lacks maturity or stability, or is it just your culture? I am asking because I am trying to imagine…
We keep customers happy because we push changes live incrementally, reduce our chances of major outages and improve our response time when they do occur.
Maybe I am just an old fuddy duddy conservative, but this struck me from the post: “In the grand scheme of things, one week isn’t that long. But to us, it felt like forever. We are constantly iterating and release multiple changes every day”. I assume they mean multiple production releases? Is this because the product lacks maturity or stability, or is it just your culture? I am asking because I am trying to imagine…
Maybe I am just an old fuddy duddy conservative, but this struck me from the post: “In the grand scheme of things, one week isn’t that long. But to us, it felt like forever. We are constantly iterating and release multiple changes every day”. I assume they mean multiple production releases? Is this because the product lacks maturity or stability, or is it just your culture? I am asking because I am trying to imagine…
Am I a curmudgeon? Not to take away from this cool writeup, but I'm familiar with a few CI/CD tools, particularly QuickBuild, Jenkins and Spinnaker. So this jumped out at me: > Our CI process was pretty standard: Every commit in an MR triggered a GitLab Pipeline, which consisted of several jobs. me: nodding silently > Those jobs would run in an auto-scaling Kubernetes cluster with up to 21 nodes me: what the actual d…
Am I a curmudgeon? Not to take away from this cool writeup, but I'm familiar with a few CI/CD tools, particularly QuickBuild, Jenkins and Spinnaker. So this jumped out at me: > Our CI process was pretty standard: Every commit in an MR triggered a GitLab Pipeline, which consisted of several jobs. me: nodding silently > Those jobs would run in an auto-scaling Kubernetes cluster with up to 21 nodes me: what the actual d…
Someone's got a new project for Q2 if they aren't doing this already - it's a pretty easy sell if you calculate out the time savings for developers during busy time of day + savings on spinning down compute resources in the middle of the night/weekends, and being able to put "I saved the company $X in idle compute and saved developers Y hours per day" on your yearly performance review looks pretty good.
Am I a curmudgeon? Not to take away from this cool writeup, but I'm familiar with a few CI/CD tools, particularly QuickBuild, Jenkins and Spinnaker. So this jumped out at me: > Our CI process was pretty standard: Every commit in an MR triggered a GitLab Pipeline, which consisted of several jobs. me: nodding silently > Those jobs would run in an auto-scaling Kubernetes cluster with up to 21 nodes me: what the actual d…
If you have it, it’s awesome. You can get parallel execution of so much, spin up environments for each branch for QA and dynamic scans. IMO it’s the optimal use case for K8s
Also, the duty cycle on the 21 nodes needs to be low enough to justify the complexity over just buying 21 computers (or getting annual pricing on 21 VMs). You could use spot instances for the EKS nodes, but then PRs will randomly fail because their instances disappear. That wastes developer salary money and productivity.
Assuming you have a ventilated room you don't care about, you could run 21 desktop towers off of ~ two-four 120V circuits. (Or buy a rack and pay ~ 2x as much for the hardware.) 21 build hosts would cost ~$21-42K. Power is probably averaging 50W per machine (they are probably mostly idle even when running tests, since they have to download stuff.) That's about 720KWh per month. US average electrical pricing is $0.20 / kWh; punitive California rates are about $0.40. So, in the punitive case, that's $288 / month.
Running 21 machines probably requires as much annoying maintenance work as EKS, though the maintenance includes swapping bad hardware, fiddling with ethernet cables, and wearing ear protection (if a rack is involved) instead of debugging piles of yaml and AWS roles, optimizing to stay in budget, etc, etc.
Earlier quoted context omitted.
The short answer is "do as little as possible". What this means in practice is breaking down every step of CI, figuring out the dependencies for that step, and then ordering the graph of dependencies such that you start as much as possible as early as possible. This process also usually shows you where things are slow and what the critical path is. Unfortunately, doing this in most CI services is actually quite diffi…
I have found Gitlab and runners the best option here.
Additionally, there's no cache guarantees between jobs within one execution. This means that you can't reliably cache an artifact in one job, and then share it with multiple downstream jobs. It mostly works, but it's hard to debug when it doesn't, especially if the cache artifact isn't versioned.
GitLab is "fine", and has some nice usability features for basic pipelines, but it's definitely not doing anything better than the other major providers with respect to these problems.