Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modern CI is too complex and misdirected (2021)

gregoryszorc.com

Post reply on HN
Print view View original

171–180 of 207 posts

Re: Modern CI is too complex and misdirected (2021)

#171

Post by mikepurvis »

DrBazza wrote: post #117
mikepurvis wrote: post #103

Earlier quoted context omitted.

There are various things that can be a reasonable candidate for the "top level" build entrypoint, including Nix, bazel, docker bake, and probably more I'm not thinking of. They all have an entrypoint that doesn't have a ton of flags or nonsense, and operate in a pretty self contained environment that they set up and manage themselves. Overall I'm not a fan of wrapping things; if there are flags or options on the top-…

Fortunately most CI/CD systems expose an environment variable during the build so you can detect most of those situations and still write a script that runs locally on a developer box. Our wrapping is 'minimal', in that you can still run bazel build //... or cmake ... and get the same build artefacts as running: build.bash release My current company is fanatical about read-only for just about every system we have (a…

Interestingly despite being pretty hard-nosed about a lot of things, Nix does not insist on a read-only source directory at build time— the source is pulled into a read-only store path, but from there it is copied into the build sandbox, not bind-mounted.

I expect this is largely a concession to the reality that most autotools projects still expect an in-source build, not to mention Python wanting to spray pyc files and build/dist directories all over the place.

Re: Modern CI is too complex and misdirected (2021)

#172

Post by [deleted] »

merb wrote: post #141
taminka wrote: post #39

Earlier quoted context omitted.

many ppl also underestimate how capable modern hardware is: for ~10usd you could handle like a million concurrent connections with a redis cluster on a handful of VPSs...

many ppl also understimate how complex it is to satisfy uptime requirements, how to scale out local infrastructure when storage > 10/50/100tb (yeah a single disk can handle that, but what about bit rot, raid stuff, etc) is involved. it gets worse when you need more servers because your ocr process of course needs cpu x so on a beefiy machine you can handle maybe 50 high page documents. but how do you talk to other ma…

[deleted]

Re: Modern CI is too complex and misdirected (2021)

#173

Post by solatic »

OP's argument hinges too much on thinking that GitLab pipelines etc. only do CI.

The purpose of Continuous Integration is to produce the One Canonical Latest Build for a given system. Well... no surprise that there's a ton of overlap between these systems and Bazel etc. "build systems".

But GitLab pipelines etc. are also Continuous Deployment systems. You don't always need fancy ArgoCD pull-based deployments or, their precursor, Chef/Puppet were also pull-based deployments for VMs. You can just have GitLab run a deployment script that calls kubectl apply, or Capistrano, or scp and ssh systemctl restart, or whatever deploys the software for you. That's not something that makes sense as part of your build system.

Re: Modern CI is too complex and misdirected (2021)

#174

Post by ThierryAbalea »

I agree with the author that CI and build systems are really trying to solve the same core problem: efficient execution of a dependency graph. And I share the view that modern CI stacks often lack the solid foundations that tools like Bazel, Gradle, or Nx bring to build systems.

Where I differ a bit is on the "two DAGs" criticism. In practice the granularity isn’t the same: the build system encodes how to compile and test, while the CI level is more about orchestration, cloning the repo, invoking the build system, publishing artifacts. That separation is useful, though we do lose the benefits of a single unified DAG for efficiency and troubleshooting.

The bigger pain points I hear from developers are less about abstractions and more about day-to-day experience: slow performance, flakiness, lack of visibility, and painful troubleshooting. For example, GitHub Actions doesn’t let you test or debug pipelines locally, you have to push every change to the remote. The hosted runners are also underpowered, and while self-hosting sounds attractive, it quickly becomes a time sink to manage reliably at scale.

This frustration is what led me to start working on Shipfox.io. Not a new CI platform, but an attempt to fix these issues on top of GitHub Actions. We’re focused on faster runners and better visibility, aggregating CI logs, test logs, CPU and memory profiles to make failures and performance problems easier to debug.

Re: Modern CI is too complex and misdirected (2021)

#175

Post by ghthor »

jcranmer wrote: post #150
nlawalker wrote: post #144

> CI offerings like GitHub Actions and GitLab Pipelines are more products than platforms because they tightly couple an opinionated configuration mechanism (YAML files) and web UI (and corresponding APIs) on top of a theoretically generic remote execute as a service offering. For me to consider these offerings as platforms, they need to grow the ability to schedule arbitrary compute via an API, without being constrai…

> I wish the author gave more concrete examples about what kinds of workflows they want to dynamically construct and remotely execute (and why a separate step of registering the workflow up front with the service before running it is such a dealbreaker), and what a sufficiently generic and unopinionated definition schema for workflows and tasks would look like as opposed to what a service like GitHub Actions defines.…

We do this at work, it’s started off as a simple build graph that used git content hashes and some simple logic to link things together. The result being that for any given pair of commits you can calculate what changed so you can only run those tests/builds etc.

We’ve paired this with buildkite which allows uploading pipeline steps at any point during the run, so our CI pipeline is one step, that generates the rest of the pipeline and uploads that.

I’m working on open sourcing this meta-build tool as I think it is niche that has no current implementation and it is not our core business.

It can build a dependency graph across many systems (terraform, go, python, nix) by parsing from those systems what they depend on. Smashes them all together, so you can have a terraform module that depends on a go binary that embeds some python; and if you change any of it then each parts can have tasks that are run (go test/build, tf plan, pytest, and etc)

Re: Modern CI is too complex and misdirected (2021)

#176

Post by donatj »

mrweasel wrote: post #154
donatj wrote: post #59

Drone was absolutely perfect back when it was Free Software. Literally "run these commands in this docker container on these events" and basically nothing more. We ran the last fully open source version much longer than we probably should have. When they went commercial, GitHub Actions became the obvious choice, but it's just married to so much weirdness and unpredictability. Whole thing with Drone opened my eyes at…

You and I have very different workflows I think. Drone was probably least intuitive system I've ever used. The idea seems nice, until you learn that Drone pretty much can't do anything useful out of the box. Want to move an artefact between steps, to bad, can't do that (at least you couldn't when we tried it out). We ended up wrapping everything in a Docker container and back to just running a bash script. Drone had…

> Drone pretty much can't do anything useful out of the box

That's the ideal. It's not doing anything you didn't explicitly tell it to.

> We ended up wrapping everything in a Docker container and back to just running a bash script.

That's literally what drone is for

Re: Modern CI is too complex and misdirected (2021)

#177

Post by reactordev »

XorNot wrote: post #132
reactordev wrote: post #119

Earlier quoted context omitted.

The rule for CI/CD and DevOps in general is boil your entire build process down to one line: ./build.sh If you want to ship containers somewhere, do it in your build script where you check to see if you’re running in “CI”. No fancy pants workflow yamls to vendor lock yourself into whatever CI platform you’re using today, or tomorrow. Just checkout, build w/ params, point your coverage checker at it. This is also the…

The other rule is that script should run as a user. Solely on that working directory. There are too many scripts like that which start, ask for sudo and then it's off to implementing someones "great idea" about your systems network interfaces.

sudo should not be required to build software.

If there’s something you require that requires sudo, it’s a pre-build environment setup on your machine. On the host. Or wherever. It’s not part of the build. If you need credentials, get them from secrets or environment variables.

Re: Modern CI is too complex and misdirected (2021)

#178

Post by maratc »

reactordev wrote: post #119
MathMonkeyMan wrote: post #15

I remember a Rich Hickey talk where he described Datomic, his database. He said "the problem with a database is that it's over there ." By modeling data with immutable "facts" (a la Prolog), much of the database logic can be moved closer to the application. In his case, with Clojure's data structures. Maybe the the problem with CI is that it's over there . As soon as it stops being something that I could set up and r…

The rule for CI/CD and DevOps in general is boil your entire build process down to one line: ./build.sh If you want to ship containers somewhere, do it in your build script where you check to see if you’re running in “CI”. No fancy pants workflow yamls to vendor lock yourself into whatever CI platform you’re using today, or tomorrow. Just checkout, build w/ params, point your coverage checker at it. This is also the…

Yeah, that's a good rule. Except, do you want to build Debug or Release? Or maybe RelWithDebugInfo? And do you want that with sanitizers maybe? And what the sanitizers' options should be? Do you want to compile your tests too, if you want to run them later on a different machine? And what about that dependency that takes two hours to compile, maybe you just want to reuse the previous compilation of it? And if so, where to take that from? Etc. etc.

Before long, you need another script that will output the train of options to your `build.sh`.

(If Fortune 500 companies can do a one-line build with zero parameters, I suspect I'd be very bored there.)

Re: Modern CI is too complex and misdirected (2021)

#179

Post by maratc »

MortyWaves wrote: post #51
MathMonkeyMan wrote: post #15

I remember a Rich Hickey talk where he described Datomic, his database. He said "the problem with a database is that it's over there ." By modeling data with immutable "facts" (a la Prolog), much of the database logic can be moved closer to the application. In his case, with Clojure's data structures. Maybe the the problem with CI is that it's over there . As soon as it stops being something that I could set up and r…

It’s why I’ve started making CI simply a script that I can run locally or on GitHub Actions etc. Then the CI just becomes a bit of yaml that runs my script.

You must be very lucky to be in a position where you know what needs to be done before the run begins. Not everyone is in that position.

At my place, we have ~400 wall hours of testing, and my run begins by figuring out what tests should be running and what can be skipped. This depends on many factors, and the calculation of the plan already involves talking to many external systems. Once we have figured out a plan for the tests, we can understand the plan for the build. Only then we can build, and test afterwards. I haven't been able to express all of that in "a bit of yaml" so far.

Re: Modern CI is too complex and misdirected (2021)

#180

Post by reactordev »

maratc wrote: post #178
reactordev wrote: post #119

Earlier quoted context omitted.

The rule for CI/CD and DevOps in general is boil your entire build process down to one line: ./build.sh If you want to ship containers somewhere, do it in your build script where you check to see if you’re running in “CI”. No fancy pants workflow yamls to vendor lock yourself into whatever CI platform you’re using today, or tomorrow. Just checkout, build w/ params, point your coverage checker at it. This is also the…

Yeah, that's a good rule. Except, do you want to build Debug or Release? Or maybe RelWithDebugInfo? And do you want that with sanitizers maybe? And what the sanitizers' options should be? Do you want to compile your tests too, if you want to run them later on a different machine? And what about that dependency that takes two hours to compile, maybe you just want to reuse the previous compilation of it? And if so, whe…

Of course we had parameters but we never ship debug builds. Treat everything like production.

If you want to debug, docker compose or add logs and metrics to seek what you find.

Post reply on HN