Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modern CI is too complex and misdirected (2021)

gregoryszorc.com

Post reply on HN
Print view View original

91–100 of 207 posts

Re: Modern CI is too complex and misdirected (2021)

#91

Post by franga2000 »

mettamage wrote: post #85
franga2000 wrote: post #38

Earlier quoted context omitted.

Requirements are complex too. Even if you don't need to scale at all, you likely do need zero-downtime deployment, easy rollbacks, server fault tolerance, service isolation... If you put your apps into containers and throw them onto Kubernetes, you get a lot of that "for free" and in a well-known and well-tested way. Hand-rolling even one of those things, let alone all of them together, would take far too much effort…

> you likely do need zero-downtime deployment I know SaaS businesses that don't as they operate in a single country, within a single timezone and the availability needs to be during business days and business hours. > easy rollbacks Yea, I haven't seen exceptions at all on this. So yea. > server fault tolerance That really depends. Many B2B or internal apps are fine with a few hours, or even a day, of downtime. > ser…

I'm definitely curious about alternatives for getting these features without k8s. Frankly, I don't like it, but I use it because it's the easiest way I've found to get all of these features. Every deployment I've seen that didn't use containers and something like k8s either didn't have a lot of these features, implemented them with a bespoke pile of shell scripts, or a mix of both.

For context, I work in exactly that kind of "everyone in one time zone" situation and none of our customers would be losing thousands by the minute if something went down for a few hours or even a day. But I still like all the benefits of a "modern devops" approach because they don't really cost much at all and it means if I screw something up, I don't have to spend too much time unscrewing it. It took a bit more time to set up compared to a basic debian server, but then again, I was only learning it at the time and I've seen friends spin up fully production-grade Kubernetes clusters in minutes. The compute costs are also negligible in the grand scheme of things.

Re: Modern CI is too complex and misdirected (2021)

#92

Post by 0xbadcafebee »

Having two different programs that are almost the same except for one or two differences, is actually better than trying to combine them.

Why do you even have a "build system"? Why not just a shell script that runs 'cc -o foo foo.c' ? Because there are more complicated things you want to do, and it would be annoying to write out a long shell script to do them all. So you have a program ('build system') that does the complicated things for you. That program then needs a config file so you can tell the program what to do.

But you want to run that 'build system' remotely when someone does a git-push. That requires a daemon on a hosted server, authentication/authorization, a git server that triggers the job when it receives a push, it needs to store secrets and pass them to the job, it needs to run it all in a container for reliability, it needs to run the job multiple times at once for parallelism, it needs to cache to speed up the jobs, it needs to store artifacts and let you browse the results or be notified of them. So you take all that complexity, put it in its own little system ('CI system'). And you make a config file so you can tell the 'CI system' how to do all that.

Could you shove both separate sets of complex features into one tool? Sure you can. But it would make it harder to develop and maintain them, change them, replace them. Much simpler to use individual smaller components to compose a larger system, than to try to build one big, complex, perfect, all-in-one-system.

Don't believe me? There's a reason most living creatures aren't 6-foot-tall amoebas. We're systems-on-systems-on-systems-on-systems (many of which have similar features) and it works pretty well. Our biggest problem is often that our individual parts aren't composeable/replaceable enough.

Re: Modern CI is too complex and misdirected (2021)

#93

Post by MortyWaves »

j4coh wrote: post #58
MortyWaves wrote: post #51

Earlier quoted context omitted.

It’s why I’ve started making CI simply a script that I can run locally or on GitHub Actions etc. Then the CI just becomes a bit of yaml that runs my script.

Are you not worried about parallelisation in your case? Or have you solved that in another way (one big beefy build machine maybe?)

Honestly not really… sure it might not be as fast but the ability to know I can debug it and build it exactly the same way locally is worth the performance hit. It probably helps I don’t write C++, so builds are not a multi day event!

Re: Modern CI is too complex and misdirected (2021)

#94

Post by maratc »

sambuccid wrote: post #50

I'm not sure why no one mentioned it yet, but the CI tool of sourcehut ( https://man.sr.ht/builds.sr.ht/ ) simplifies all of this. It just spins a linux distro of your choice, and executes a very bare bone yml that essentially contains a lot of shell commands, so it's also easy to replicate locally. There are 12 yml keywords in total that cover everything. Other cool things are the ability to ssh in a build if it fai…

Everything I've seen that's based on yaml makes easy things trivial, and hard things impossible.

This caused me to default back to Jenkins several times already, now I'm in a position to never wander off to another yaml-based tool.

Re: Modern CI is too complex and misdirected (2021)

#95

Post by MortyWaves »

maccard wrote: post #70
MortyWaves wrote: post #51

Earlier quoted context omitted.

It’s why I’ve started making CI simply a script that I can run locally or on GitHub Actions etc. Then the CI just becomes a bit of yaml that runs my script.

How does that script handle pushing to ghcr, or pulling an artifact from a previous stage for testing? In my experience these are the bits that fail all the time, and are the most important parts of CI once you go beyond it taking 20/30 seconds to build. A clean build in an ephemeral VM of my project would take about 6 hours on a 16 core machine with 64GB RAM.

To be honest I haven’t really thought about it and it’s definitely something it can’t do, you’d probably need to call their APIs or something.

I am fortunate in that the only things I want to reuse is package manager caches.

Re: Modern CI is too complex and misdirected (2021)

#96

Post by jopsen »

donatj wrote: post #59

Drone was absolutely perfect back when it was Free Software. Literally "run these commands in this docker container on these events" and basically nothing more. We ran the last fully open source version much longer than we probably should have. When they went commercial, GitHub Actions became the obvious choice, but it's just married to so much weirdness and unpredictability. Whole thing with Drone opened my eyes at…

It's never just running commands in a container.

Don't get me wrong, it's a fantastic primitive.

But eventually you need to conditionally run some tests (to save compute).

For some benchmarks you might have limited hardware, so you need to coalesce jobs, and only run every 5 or 10 commits. You might want to keep the hardware hot, but also the queue small. So ideally you want to coalesce dynamically.

You also want result reporting, comparisons to previous results. Oh, and since you're coalescing some jobs and doing others conditionally you'll need ways to manually trigger skipped jobs later, maybe bisect too.

It's when you need to economize your compute that CI can get really complex. Especially, if you have fragile benchmark that or flaky tests.

Yes, in theory you can enforce a culture that removes flaky tests, but doing so often requires tooling support -- statistics, etc.

Re: Modern CI is too complex and misdirected (2021)

#97

Post by gchamonlive »

nrclark wrote: post #89
DrBazza wrote: post #64

Earlier quoted context omitted.

Your build should be this: build.bash and that's it (and that can even trigger a container build). I've spent far too much time debugging CI builds that work differently to a local build, and it's always because of extra nonsense added to the CI server somehow. I've yet to find a build in my industry that doesn't yield to this 'pattern'. Your environment setup should work equally on a local machine or a CI/CD server,…

Agreed with this sentiment, but with one minor modification: use a Makefile instead. Recipes are still chunks of shell, and they don’t need to produce or consume any files if you want to keep it all task-based. You get tab-completion, parallelism, a DAG, and the ability to start anywhere on the task graph that you want. It’s possible to do all of this with a pure shell script, but then you’re probably reimplementing…

Just be aware of the "Makefile effect"[1] which can easily devolve into the Makefile also being "over there", far from the application, just because it's actually a patchwork of copy-paste targets stitched together.

[1] https://news.ycombinator.com/item?id=42663231

Re: Modern CI is too complex and misdirected (2021)

#98

Post by pointlessone »

I can’t say I like OP’s vision. My main objection is that this vision is terminally online. I want to be able to run the whole build locally (for when my internet is down, or I’m on a plane, or on a remote island in a cave, etc.). The local build and CI should only differ in that local build is triggered manually and results are reported in the terminal (or IDE) and CI build is triggered by a push and reported on the PR (or other web page, or API endpoint, etc. ). It should be the same but for the entry and exit. Tasks, queues, DAGs, etc. it’s all nice but ultimately are implementation details. Even make has DAGs, tasks, and parallel execution. Unless the build can run locally it’s as if there’s no build. Differences between local build and CI, be it because of environment, tasks setup, caching, whatever makes CI painful. It’s precisely because you have a build system for local builds and a separate CI setup that the world contains 10% more misery than it should.

So basically either the whole CI pipeline is just a single command invoking my build system or the CI pipeline can be ran locally. Any other arrangement is self-inflicted suffering.

Re: Modern CI is too complex and misdirected (2021)

#99

Post by dgfitz »

nrclark wrote: post #89
DrBazza wrote: post #64

Earlier quoted context omitted.

Your build should be this: build.bash and that's it (and that can even trigger a container build). I've spent far too much time debugging CI builds that work differently to a local build, and it's always because of extra nonsense added to the CI server somehow. I've yet to find a build in my industry that doesn't yield to this 'pattern'. Your environment setup should work equally on a local machine or a CI/CD server,…

Agreed with this sentiment, but with one minor modification: use a Makefile instead. Recipes are still chunks of shell, and they don’t need to produce or consume any files if you want to keep it all task-based. You get tab-completion, parallelism, a DAG, and the ability to start anywhere on the task graph that you want. It’s possible to do all of this with a pure shell script, but then you’re probably reimplementing…

You invoke CMake/qmake/configure/whatever from the bash script.

I hate committing makefiles directly if it can be helped.

You can still call make in the script after generating the makefile, and even pass the make target as an argument to the bash script if you want. That being said, if you’re passing more than 2-3 arguments to the build.sh you’re probably doing it wrong.

Re: Modern CI is too complex and misdirected (2021)

#100

Post by nrclark »

dgfitz wrote: post #99
nrclark wrote: post #89

Earlier quoted context omitted.

Agreed with this sentiment, but with one minor modification: use a Makefile instead. Recipes are still chunks of shell, and they don’t need to produce or consume any files if you want to keep it all task-based. You get tab-completion, parallelism, a DAG, and the ability to start anywhere on the task graph that you want. It’s possible to do all of this with a pure shell script, but then you’re probably reimplementing…

You invoke CMake/qmake/configure/whatever from the bash script. I hate committing makefiles directly if it can be helped. You can still call make in the script after generating the makefile, and even pass the make target as an argument to the bash script if you want. That being said, if you’re passing more than 2-3 arguments to the build.sh you’re probably doing it wrong.

Yes to calling CMake/etc. No to checking in generated Makefiles. But for your top-level “thing that calls CMake”, try writing a Makefile instead of a shell script. You’ll be surprised at how powerful it is. Make is a dark horse.
Post reply on HN