Jenkins Is Getting Old

Also, if you ever need to schedule jobs/tasks (not just ci builds) across multiple machines, buildbot is great because all you need is a master, and slave python processes which just need a network connection to the master.

Concourse really hits his requirements for ops-friendliness and testability. It's easy to upgrade because the web host and CI workers are completely stateless, and the work you do with Concourse is easy to test because the jobs themselves are all completely self-contained. Because Concourse forces you to factor your builds into inputs, tasks, and outputs, it becomes somewhat straightforward to test your outputs by replacing them with mock outputs.

The main issue with Concourse is that it has a huge learning curve to use effectively. When you have many pipelines, you end up learning the undocumented meta pipeline pattern to help manage them. You end up implementing stuff like mock outputs by yourself, since it's not really a first-class concept. Concepts like per-branch versioning that have been in products like Jenkins for years are only now entering development as "spaces". All of the configuration can be version controlled, but it's all YAML, so to wrangle everything together, you end up using something which will compile down to YAML instead. RBAC much improved in v5 but still leaves much to be desired. There are no manual steps, only manual jobs. Job triggering is relatively slow due to a fundamental design tradeoff where webhooks trigger resource checks which trigger jobs instead of triggering jobs directly, to make resources easier to maintain. Nobody really tells you this on the front page.

It has its flaws. But if you delve into it you see very quickly that it's an incredibly solid product. The ability to run one-off jobs from developer workstations on CI workers, and to easily SSH into those CI worker containers from developer workstations, is incredibly helpful for debugging issues. Because everything is stateless, everything tends to "just work". If you go in with your eyes open about its limitations, you'll have a good time.

First you checkout the project from the repo and it just works, doesn't matter GIT, SVN or whatever. How many plugins does it take to checkout a project in jenkins? Is there even a git plugin working nowadays?

Then, you build the project. If it's any of C# or Java for example, the ant/maven/sln/nuget files are detected automatically, just click next and it's built. Does jenkins even understand what is a requirements.txt? Hint: It's not a bash script.

The JVM and the Visual Studio are detected automatically on all the build slaves and the project is already building in the right places. If you want to aim for specific tool versions, there are presets variables on all hosts to filter where to build and to use in build scripts so paths are always right. How is the build matrix plugin in Jenkins lately? Broken as usual?

The project is built, or is it building? It's easy to tell because there is a clear colored status icon and the estimated time to completion is displayed. Teamcity offers that out of the box for maybe 15 years now. Well, jenkins finally got a progress bar too a couple years ago. I guess I'm defeated, Jenkins caught up on basic core functionality only a decade late, I can't justify to pay for working and polished tools anymore. Well, I hope our sysadmin will install the Extra Status Icon Plugin or we'll have to live without the big colored circles next to the build.

Personally I’ve found that for my past and present use cases generating any needed step (e.g test matrix) e.g with a script is much more flexible, predictable, and reproducible since the generated result can be versioned.

I also successfully used various custom runners including baremetal Windows ones and virtualised macOS ones inside VirtualBox.