Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modern CI is too complex and misdirected (2021)

gregoryszorc.com

Post reply on HN
Print view View original

121–130 of 207 posts

Re: Modern CI is too complex and misdirected (2021)

#121

Post by MathMonkeyMan »

dapperdrake wrote: post #44
MathMonkeyMan wrote: post #15

I remember a Rich Hickey talk where he described Datomic, his database. He said "the problem with a database is that it's over there ." By modeling data with immutable "facts" (a la Prolog), much of the database logic can be moved closer to the application. In his case, with Clojure's data structures. Maybe the the problem with CI is that it's over there . As soon as it stops being something that I could set up and r…

Transactions and a single consistent source of truth with stuff like observability and temporal ordering is centralized and therefore "over there" for almost every place you could be in. As long as communications have bounded speed (speed of light or whatever else) there will be event horizons. The point of a database is to track changes and therefore time centrally . Not because we want to, but because everything el…

I haven't used Datomic, but you're right that the part that requires over there is "single consistent source of truth." There's only ever a single node that is sequencing all writes. Perhaps as a result of this, it provides strong [verified ACID guarantees][1].

What I got from Hickey's talk is that he wanted to design a system that resisted the urge to encode everything in a stored procedure and run it on the database server.

[1]: https://jepsen.io/analyses/datomic-pro-1.0.7075

Re: Modern CI is too complex and misdirected (2021)

#122

Post by esafak »

The code-based CI platform dagger.io used to support CUE lang but dropped it due to lack of interest. Combining that with something like bazel, all in CUE or Skylark sounds interesting, but bazel and dagger are both pretty complex on their own. Their merger would be too much.

Re: Modern CI is too complex and misdirected (2021)

#123

Post by kqr »

DrBazza wrote: post #64
MathMonkeyMan wrote: post #15

I remember a Rich Hickey talk where he described Datomic, his database. He said "the problem with a database is that it's over there ." By modeling data with immutable "facts" (a la Prolog), much of the database logic can be moved closer to the application. In his case, with Clojure's data structures. Maybe the the problem with CI is that it's over there . As soon as it stops being something that I could set up and r…

Your build should be this: build.bash and that's it (and that can even trigger a container build). I've spent far too much time debugging CI builds that work differently to a local build, and it's always because of extra nonsense added to the CI server somehow. I've yet to find a build in my industry that doesn't yield to this 'pattern'. Your environment setup should work equally on a local machine or a CI/CD server,…

I tried to drive this approach at a previous job but nobody else on the team cared so I ended up always having to mirror all the latest build changes into my bash script.

The reason it didn't catch on? Everyone else was running local builds in a proprietary IDE, so to them the local build was never the same anyway.

Re: Modern CI is too complex and misdirected (2021)

#124

Post by mike_hearn »

ohdeargodno wrote: post #25
donperignon wrote: post #16

2025 and Jenkins still the way to go

The fact that maintaining any Jenkins instance makes you want to shoot yourself and yet it's the least worst option is an indictment of the whole CI universe. I have never seen a system with documentation as awful as Jenkins, with plugins as broken as Jenkins, with behaviors as broken as Jenkins. Groovy is a cancer, and the pipelines are half assed, unfinished and incompatible with most things.

Least worst compared to what? You think TeamCity is worse?

Re: Modern CI is too complex and misdirected (2021)

#125

Post by pxc »

reactordev wrote: post #119
MathMonkeyMan wrote: post #15

I remember a Rich Hickey talk where he described Datomic, his database. He said "the problem with a database is that it's over there ." By modeling data with immutable "facts" (a la Prolog), much of the database logic can be moved closer to the application. In his case, with Clojure's data structures. Maybe the the problem with CI is that it's over there . As soon as it stops being something that I could set up and r…

The rule for CI/CD and DevOps in general is boil your entire build process down to one line: ./build.sh If you want to ship containers somewhere, do it in your build script where you check to see if you’re running in “CI”. No fancy pants workflow yamls to vendor lock yourself into whatever CI platform you’re using today, or tomorrow. Just checkout, build w/ params, point your coverage checker at it. This is also the…

You still inevitably need a bunch of CI platform-specific bullshit for determining "is this a pull request? which branch am I running on?", etc. Depending on what you're trying to do and what tools you're working with, you may need such logic both in an accursed YAML DSL and in your build script.

And if you want your CI jobs to do things like report cute little statuses, integrate with your source forge's static analysis results viewer, or block PRs, you have to integrate with the forge at a deeper level.

There aren't good tools today for translating between the environment variables or other things that various CI platforms expose, managing secrets (if you use CI to deploy things) that are exposed in platform-specific ways, etc.

If all you're doing with CI is spitting out some binaries, sure, I guess. But if you actually ask developers what they want out of CI, it's typically more than that.

Re: Modern CI is too complex and misdirected (2021)

#126

Post by SideburnsOfDoom »

qwertytyyuu wrote: post #6

Wait a CI isn't supposed to be a build system that also runs tests?

> a CI isn't supposed to be a build system?

No. "Continuous Integration" is the practice of frequently merging changes to main. In this sense, "integration" means to take my changes and combine them with other recent changes.

A build and test system like those described in this article is a way to make CI safe and fast. It's not CI itself, it's just the enabling automation: the pre-merge checks and the post-merge artefact creation.

Re: Modern CI is too complex and misdirected (2021)

#127

Post by HPsquared »

reactordev wrote: post #119
MathMonkeyMan wrote: post #15

I remember a Rich Hickey talk where he described Datomic, his database. He said "the problem with a database is that it's over there ." By modeling data with immutable "facts" (a la Prolog), much of the database logic can be moved closer to the application. In his case, with Clojure's data structures. Maybe the the problem with CI is that it's over there . As soon as it stops being something that I could set up and r…

The rule for CI/CD and DevOps in general is boil your entire build process down to one line: ./build.sh If you want to ship containers somewhere, do it in your build script where you check to see if you’re running in “CI”. No fancy pants workflow yamls to vendor lock yourself into whatever CI platform you’re using today, or tomorrow. Just checkout, build w/ params, point your coverage checker at it. This is also the…

Sounds like the Lotus philosophy, "simplify and add lightness".

Re: Modern CI is too complex and misdirected (2021)

#128

Post by germandiago »

I am running buildbot with a customized matrix style buildbot for years for my side projects.

This is because yes, it is very complex. I have tried Jenkins before and Gitlab CI.

Something that most build tools and CIs should learn from Meson build system is that sometimes it is better to just keep it simple than adding features on top. If you need them, script them in some way but keep configuration as data-driven (and I mean purely data-driven, not half a language).

My build system is literally: a build matrix, where you can specify filters of what to keep or skip. This gets all combined.

A series of steps with a name that can be executed or not depending on a filter. Nothing else. Every step calls the build system or whatever.

After that it sends mail reports and integrates with Gerrit to send builds and Gerrit csn also csll it.

No fsncy plugins or the like. Just this small toml file I have and run normal scripts or command lines without 300 layers on top. There are already enough things that can break so that one keeps adding opaque layers on top. Just use the tools we all know: ssh, bash, Python etc.

Everyone knows how to call that. If a step is too complex, just make a script.

Re: Modern CI is too complex and misdirected (2021)

#129

Post by germandiago »

DrBazza wrote: post #64
MathMonkeyMan wrote: post #15

I remember a Rich Hickey talk where he described Datomic, his database. He said "the problem with a database is that it's over there ." By modeling data with immutable "facts" (a la Prolog), much of the database logic can be moved closer to the application. In his case, with Clojure's data structures. Maybe the the problem with CI is that it's over there . As soon as it stops being something that I could set up and r…

Your build should be this: build.bash and that's it (and that can even trigger a container build). I've spent far too much time debugging CI builds that work differently to a local build, and it's always because of extra nonsense added to the CI server somehow. I've yet to find a build in my industry that doesn't yield to this 'pattern'. Your environment setup should work equally on a local machine or a CI/CD server,…

I always use, no matter what I am using underneath, a bootstrap script, a configure script and a build step.

That keeps the cli interface easy, expectable and guessable.

Re: Modern CI is too complex and misdirected (2021)

#130

Post by dgfitz »

nrclark wrote: post #100
dgfitz wrote: post #99

Earlier quoted context omitted.

You invoke CMake/qmake/configure/whatever from the bash script. I hate committing makefiles directly if it can be helped. You can still call make in the script after generating the makefile, and even pass the make target as an argument to the bash script if you want. That being said, if you’re passing more than 2-3 arguments to the build.sh you’re probably doing it wrong.

Yes to calling CMake/etc. No to checking in generated Makefiles. But for your top-level “thing that calls CMake”, try writing a Makefile instead of a shell script. You’ll be surprised at how powerful it is. Make is a dark horse.

I wouldn't be surprised at all, make is great!

My contention is that a build script should ideally be:

sha-bang

clone && cd $cloned_folder

${generate_makefile_with_tool}

make $1

Anything much longer than that can (and usually will) quickly spiral out of control.

Make is great. Unless you're code-golfing, your makefile will be longer than a few lines and a bunch of well-intentioned-gremlins will pop in and bugger the whole thing up. Just seen it too many times.

Edit: in the jenkins case, in a jenkins build shell the clone happens outside build.sh:

(in jenkins shell):

clone && cd clone ./build.sh $(0-1 args)

(inside build.sh): $(generate_makefile_with_tool) make $1

Post reply on HN