Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modern CI is too complex and misdirected (2021)

gregoryszorc.com

Post reply on HN
Print view View original

191–200 of 207 posts

Re: Modern CI is too complex and misdirected (2021)

#191

Post by lenkite »

dgfitz wrote: post #99
nrclark wrote: post #89

Earlier quoted context omitted.

Agreed with this sentiment, but with one minor modification: use a Makefile instead. Recipes are still chunks of shell, and they don’t need to produce or consume any files if you want to keep it all task-based. You get tab-completion, parallelism, a DAG, and the ability to start anywhere on the task graph that you want. It’s possible to do all of this with a pure shell script, but then you’re probably reimplementing…

You invoke CMake/qmake/configure/whatever from the bash script. I hate committing makefiles directly if it can be helped. You can still call make in the script after generating the makefile, and even pass the make target as an argument to the bash script if you want. That being said, if you’re passing more than 2-3 arguments to the build.sh you’re probably doing it wrong.

I have experienced horror build systems where the Makefile delegates to a shell script which then delegates to some sub-module Makefile, which then delegates to a shell script...

The problem is that shell commands are very painful to specify in a Makefile with weird syntactical rules. Esp when you need them to run in one shell - a lot of horror quoting needed.

Re: Modern CI is too complex and misdirected (2021)

#192

Post by jiggawatts »

arunix wrote: post #142
bob1029 wrote: post #32

I've been able to effectively skip the entire CI/CD conversation by preferring modern .NET and SQLite. I recently spent a day trying to get a GH Actions build going but got frustrated and just wrote my own console app to do it. Polling git, tracking a commit hash and running dotnet build is not rocket science. Putting this agent on the actual deployment target skips about 3 boss fights.

Is there something about .NET that makes this easier?

It's like Java in that it tends towards the "build once, run anywhere" style.

Also, Windows has a consistent user-mode API surface (unlike Linux), so a .NET app that runs on a desktop will run on server almost always.

The same cannot be said for someone developing on a "UNIX-like" system such a MacOS and then trying to run it on Ubuntu... or RedHat. Alpine? Shit...

Re: Modern CI is too complex and misdirected (2021)

#193

Post by BobbyTables2 »

reactordev wrote: post #119
MathMonkeyMan wrote: post #15

I remember a Rich Hickey talk where he described Datomic, his database. He said "the problem with a database is that it's over there ." By modeling data with immutable "facts" (a la Prolog), much of the database logic can be moved closer to the application. In his case, with Clojure's data structures. Maybe the the problem with CI is that it's over there . As soon as it stops being something that I could set up and r…

The rule for CI/CD and DevOps in general is boil your entire build process down to one line: ./build.sh If you want to ship containers somewhere, do it in your build script where you check to see if you’re running in “CI”. No fancy pants workflow yamls to vendor lock yourself into whatever CI platform you’re using today, or tomorrow. Just checkout, build w/ params, point your coverage checker at it. This is also the…

You’re not wrong but your suggestion also throws away a lot of major benefits of CI. I agree jobs should be one liners but we still need more than one…

The single job pipeline doesn’t tell you what failed. It doesn’t parallelize unit and integration test suites while dealing with the combinatorial matrix of build type, target device, etc.

At some point, a few CI runners become more powerful than a developer’s workstation. Parallelization can really matter for reducing CI times.

I’d argue the root of the problem is that we are stuck on using “make” and scripts for local build automation.

We need something descriptive enough to describe a meaningful CI pipeline but also allow local execution.

Sure, one can develop a bespoke solution, but reinventing the wheel each time gets tiring and eventually becomes a sizable time sink.

In principle, we should be able to execute pieces of .gitlab-ci.yml locally, but even that becomes non trivial with all the nonstandard YAML behaviors done in gitlab, not to mention the varied executor types.

Instead we have a CI workflow and a local workflow and hope the two are manually kept in sync.

In some sense, the current CI-only automation tools shouldn’t even need to exist (gitlab, Jenkins, etc) — why didn’t we just use a cron job running “build.sh” ?

I argue these tools should mainly only have to focus on the “reporting/artifacts” with the pipeline execution parts handled elsewhere (or also locally for a developer).

Shame on you GitLab!

Re: Modern CI is too complex and misdirected (2021)

#194

Post by rirze »

e1gen-v wrote: post #65

I’ve been using Pulumi automation in our CI and it’s been really nice. There’s definitely a learning curve with the asynchronous Outputs but it’s really nice for building docker containers and separating pieces of my infra that may have different deployment needs.

This comment is specifically about using CI/CD to deploy infrastructure, right? Or does Pulumi have a framework for general CI/CD?

Re: Modern CI is too complex and misdirected (2021)

#195

Post by wredcoll »

pxc wrote: post #189
wredcoll wrote: post #185

Earlier quoted context omitted.

My experience with nix, at a smaller scale than what you're talking about, is that it only worked as long as every. single. thing. was reimplemented inside nix. Once one thing was outside of nix, everything exploded and writing a workaround was miserable because the nix configuration did not make it easy.

> every. single. thing. was reimplemented inside nix That's kinda what hermetic means, though, isn't it? Whether that's painful or not, that's pretty much exactly what GGP was asking for! > Once one thing was outside of nix, everything exploded and writing a workaround was miserable because the nix configuration did not make it easy. Nix doesn't make it easy to have Nix builds depend on non-Nix things (this is requir…

It was the dev environment for a bunch of wannabe microservices running across node/java/python

And like, I'm getting to the point of being old enough that I've "seen this before"; I feel like I've seen other projects that went "this really hard problem will be solved once we just re-implement everything inside our new system" and it rarely works; you really need a degree of pragmatism to interact with the real world. Systemd and Kubernetes are examples of things that do a lot of re-implementation but are mostly better than the previous.

Re: Modern CI is too complex and misdirected (2021)

#196

Post by reactordev »

merb wrote: post #137
michaelmior wrote: post #133

Earlier quoted context omitted.

A lot of CI platforms (such as GitHub) spit out a lot of environment variables automatically that can help you with the logic in your build script. If they don't, they should give you a way to set them. One approach is to keep the majority of the logic in your build script and just use the platform-specific stuff to configure the environment for the build script. Of course, as you mention, if you want to do things li…

how can you build your containers in parallel? over multiple machines? I'm not sure that a sh script can do that with github

these problems are general ones and the solution is the same as running programs in parallel or across machines. When needing to build different architectures (and needing a host to provide the toolchains), what's stopping you from issuing more than 1 command in your CI/CD pipeline? Most pipelines have a way of running something on a specific host. So does k8s, ecs, , and probably your IT team.

My experience, when it gets time to actually build the thing. A one-liner (with args if you need them) is the best approach. If you really REALLY need to, you can have more than one script for doing it - depending on what path down the pipeline you take. Maybe it's

    1) ./build.sh -config Release
    2) ./deploy.sh -docker -registry= --kick
Just try not to go too crazy. The larger the org, the larger this wrangling task can be. Look at Google and gclient/gn. Not saying it's bad, just saying it's complicated for a reason. You don't need that (you'll know if you do).

The point I made is I hate when I see 42 lines in a build workflow yaml that isn't syntax highlighted because it's been |'d in there. I think the yaml's of your pipelines, etc, should be configuration for the pipeline and the actual execution should be outsourced to a script you provide.

Re: Modern CI is too complex and misdirected (2021)

#197

Post by reactordev »

BobbyTables2 wrote: post #193
reactordev wrote: post #119

Earlier quoted context omitted.

The rule for CI/CD and DevOps in general is boil your entire build process down to one line: ./build.sh If you want to ship containers somewhere, do it in your build script where you check to see if you’re running in “CI”. No fancy pants workflow yamls to vendor lock yourself into whatever CI platform you’re using today, or tomorrow. Just checkout, build w/ params, point your coverage checker at it. This is also the…

You’re not wrong but your suggestion also throws away a lot of major benefits of CI. I agree jobs should be one liners but we still need more than one… The single job pipeline doesn’t tell you what failed. It doesn’t parallelize unit and integration test suites while dealing with the combinatorial matrix of build type, target device, etc. At some point, a few CI runners become more powerful than a developer’s worksta…

You are mistaking a build for a pipeline. I still believe in pipelines and configuring the right hosts/runners to produce your artifacts. Your actual build on that host/runner should be a one-liner.

Re: Modern CI is too complex and misdirected (2021)

#198

Post by reactordev »

mrbombastic wrote: post #145
reactordev wrote: post #119

Earlier quoted context omitted.

The rule for CI/CD and DevOps in general is boil your entire build process down to one line: ./build.sh If you want to ship containers somewhere, do it in your build script where you check to see if you’re running in “CI”. No fancy pants workflow yamls to vendor lock yourself into whatever CI platform you’re using today, or tomorrow. Just checkout, build w/ params, point your coverage checker at it. This is also the…

How do you get caching of build steps with this approach? Or do you just not?

We get them with docker.

Everything becomes a container so why not use the container engine for it. If you know how layers work…

Re: Modern CI is too complex and misdirected (2021)

#199

Post by reactordev »

immibis wrote: post #184
reactordev wrote: post #177

Earlier quoted context omitted.

sudo should not be required to build software. If there’s something you require that requires sudo, it’s a pre-build environment setup on your machine. On the host. Or wherever. It’s not part of the build. If you need credentials, get them from secrets or environment variables.

For use cases like making tar files with contents owned by root, Debian developed the tool "fakeroot", which intercepts standard library functions so that when the build script sets a file to be owned by root and then reads the ownership later, it sees it's owned by root, so it records that in the tar file.

Debian takes the You can’t touch this approach to things to solve their issues. Instead of work arounds, they just hack at the lower kernel level and trace all you do. It’s a flex. fakeroot isn’t the only tool like this. I love me some Debian.

Re: Modern CI is too complex and misdirected (2021)

#200

Post by pxc »

wredcoll wrote: post #195
pxc wrote: post #189

Earlier quoted context omitted.

> every. single. thing. was reimplemented inside nix That's kinda what hermetic means, though, isn't it? Whether that's painful or not, that's pretty much exactly what GGP was asking for! > Once one thing was outside of nix, everything exploded and writing a workaround was miserable because the nix configuration did not make it easy. Nix doesn't make it easy to have Nix builds depend on non-Nix things (this is requir…

It was the dev environment for a bunch of wannabe microservices running across node/java/python And like, I'm getting to the point of being old enough that I've "seen this before"; I feel like I've seen other projects that went "this really hard problem will be solved once we just re-implement everything inside our new system" and it rarely works; you really need a degree of pragmatism to interact with the real world…

> Systemd and Kubernetes are examples of things that do a lot of re-implementation but are mostly better than the previous.

I feel the same way about systemd, and I'll take your word for it with respect to Kubernetes. :)

> "this really hard problem will be solved once we just re-implement everything inside our new system" [...] rarely works

Yes. 100%. And this is definitely characteristic of Nix's ambition in some ways as well as some of the most painful experiences users have with it.

> you really need a degree of pragmatism to interact with the real world

Nix is in fact founded on a huge pragmatic compromise: instead of beginning with a new operating system, or a new executable format with a new linker, or even a new basic build system (a la autotools or make)! Instead of doing any of those things, Nix's design manages to bring insights and features from programming language design (various functional programming principles and, crucially, memoization and garbage collection) to build systems and package management tools, on top of existing (even aging) operating systems and toolchains.

I would also contend that the Nixpkgs codebase is a treasure, encoding how to build, run, and manage an astonishing number of apps (over 120,000 packages now) and services (I'd guess at least 1,000; there are some 20,000 configuration options built into NixOS). I think this does to some extent demonstrate the viability of getting a wide variety of software to play nice with Nix's commitments.

Finally, and it seems you might not be aware of this, but there are ways within Nix to relax the normal constraints! And of course you can also use Nix in various ways without letting Nix run the show.[0] (I'm happy to chat about this. My team, for instance, uses Nix to power Python development environments for AWS Lambdas without putting Nix in charge of the entire build process.)

However:

  - fully leveraging Nix's benefits requires fitting within certain constraints
  - the Nix community, culturally, does not show much interest in relaxing those constraints even when possible[1], but there is more and more work going on in this area in recent years[2][3] and some high-profile examples/guides of successful gradual adoption[4]
  - the Node ecosystem's habit of expecting arbitrary network access at build time goes against one of the main constraints that Nix commits to by default, and *this indeed often makes packaging Node projects "properly" with Nix very painful*
  - Python packaging is a mess and Nix does help IME, but getting there can be painful
Maybe if you decide to play with Nix again, or you encounter it on a future personal or professional project, you can remember this and look for ways to embrace the "heretical" approach. It's more viable and more popular than ever :)

--

0: https://zimbatm.com/notes/nix-packaging-the-heretic-way ; see also the community discussion of the post here: https://discourse.nixos.org/t/nix-packaging-the-heretic-way/...

1: See Graham Christensen's 2022 NixCon talk about this here. One such constraint he discusses relaxing, build-time sandboxing, is especially useful for troublesome cases some Node projects: https://av.tib.eu/media/61011

2: See also Tom Bereknyei's NixCon talk from the same year; the last segment of it is representative of increasing interest among technical leaders in the Nix community on better enabling and guiding gradual adoption: https://youtu.be/2iugHjtWqIY?t=830

3: Towards enabling gradual adoption for the most all-or-nothing part of the Nix ecosystem, NixOS, a talk by Pierre Penninckx from 2024: https://youtu.be/CP0hR6w1csc

4: One good example of this is Mitchell Hashimoto's blog posts on using Nix with Dockerfiles, as opposed to the purist's approach of packaging your whole environment via Nix and then streaming the Nix packages to a Docker image using a Nix library like `dockerTools` from Nixpkgs: https://mitchellh.com/writing/nix-with-dockerfiles

Post reply on HN