Modern CI is too complex and misdirected (2021)

I wrote Linci to tackle this issue a few years back

Https://linci.tp23.org

Ci is too complicated and are basically about locking. But what you (should) do is run cli commands on dedicated boxes in remote locations.

In Linci every thing done remote is the same locally. Just pick a box for the job.

There is almost no code, and what there is could be rewritten is any language if you prefer. Storage is git/VCs + filesystem.

Filesystem are kit fashionable because they are a problem for the big boys but not for you or I. File system storage makes thing easy and hackable.

That is unix bread and butter. Microsoft need a ci in yaml. Linux does not.

Been using it for a while an a small scale and it's never made me want anything else.

Scripting bash Remoting ssh Auth pam Notification irc/II (Or mail stomp etc) Scheduling crond Webhooks not needed if repo is on the same container use bash for most hooks, and nodejs server that calls cli for github

Each and every plug-in is a bash script and some env variables.

Read other similar setups hacked up with make. But I don't like the env vars handling and syntax of make. Bash is great if what you do is simple, and as the original article points out so clearly, if your ci is complicated you should probably rethink it.

The most concerning part about modern CI to me is how most of it is running on GitHub Actions, and how GitHub itself has been deprioritizing GitHub Actions maintenance and improvements over AI features.

Seriously, take a look at their pinned repo: https://github.com/actions/starter-workflows

> Thank you for your interest in this GitHub repo, however, right now we are not taking contributions.

> We continue to focus our resources on strategic areas that help our customers be successful while making developers' lives easier. While GitHub Actions remains a key part of this vision, we are allocating resources towards other areas of Actions and are not taking contributions to this repository at this time.

This is pretty much my experience too. Working with jenkins is always complete pain, but at the same time I can't identify any really solid alternatives either. So far sourcehut builds is looking the most promising, but I haven't had chance to use it seriously. While it's nominally part of the rest of sourcehut ecosystem, I believe it could be run with minor tweaks also standalone if needed

I am working on this problem and while I agree with the author, there is room for improvement for the current status quo:

> So going beyond the section title: CI systems aren't too complex: they shouldn't need to exist. Your CI functionality should be an extension of the build system.

True. In the sense that if you are running a test/build, you probably want to start local first (dockerize) and then run that container remotely. However, the need for CI stems from the fact that you need certain variables (ie: you might want to run this, when commit that or pull request this or that, etc.) In a sense, a CI system goes beyond the state of your code to the state of your repo and stuff connected to your repo (ie: slack)

> There is a GitHub Actions API that allows you to interact with the service. But the critical feature it doesn't let me do is define ad-hoc units of work: the actual remote execute as a service. Rather, the only way to define units of work is via workflow YAML files checked into your repository. That's so constraining!

I agree. Which is why most people will try to use the container or build system to do these complex tasks.

> Taskcluster's model and capabilities are vastly beyond anything in GitHub Actions or GitLab Pipelines today. There's a lot of great ideas worth copying.

You still need to run these tasks as containers. So, say if you want to compare two variables, that's a lot of compute for a relatively simple task. Which is why the status quo has settled with GitHub Actions.

> it should offer something like YAML configuration files like CI systems do today. That's fine: many (most?) users will stick to using the simplified YAML interface.

It should offer a basic programming/interpreted language like JavaScript.

This is an area where WebAssembly can be useful. At its core, WASM is a unit of execution. It is small, universal, cheap and has a very fast startup time compared to a full OS container. You can also run arbitrarily complex code in WASM while ensuring isolation.

My idea here is that CI becomes a collection of executable tasks that the CI architect can orchestrate while the build/test systems remain a simple build/test command that run on a traditional container.

> Take Mozilla's Taskcluster and its best-in-class specialized remote execute as a service platform.

That would be a mistake, in my opinion. There is a reason Taskcluster has failed to get any traction. Most people are not interested in engineering their CI but in getting tasks executed on certain conditions. Most companies don't have people/teams dedicated for this and it is something developers do alongside their build/test process.

> Will this dream become a reality any time soon? Probably not. But I can dream. And maybe I'll have convinced a reader to pursue it.

I am :) I do agree with your previous statement that it is a hard market to crack.