Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modern CI is too complex and misdirected (2021)

gregoryszorc.com

Post reply on HN
Print view View original

111–120 of 207 posts

Re: Modern CI is too complex and misdirected (2021)

#111

Post by franga2000 »

stonemetal12 wrote: post #108
franga2000 wrote: post #91

Earlier quoted context omitted.

I'm definitely curious about alternatives for getting these features without k8s. Frankly, I don't like it, but I use it because it's the easiest way I've found to get all of these features. Every deployment I've seen that didn't use containers and something like k8s either didn't have a lot of these features, implemented them with a bespoke pile of shell scripts, or a mix of both. For context, I work in exactly that…

>I use it because it's the easiest way I've found to get all of these features. Every deployment I've seen that didn't use containers and something like k8s either didn't have a lot of these features, implemented them with a bespoke pile of shell scripts, or a mix of both. Features aren't pokemon you don't have to catch them all. Back when stackoverflow was cool and they talked about their infrastructure, they were r…

A) Not much more. The per-node overhead is relatively small and it's not unlikely that they could have made some efficiency gains by having a homogenous cluster that saved them some nodes to offset that.

B) Why on earth would you need to do that? K8s is, at its core, just a thing that runs containers. Take your existing app, stick it in a container and write a little yaml explaining which other containers it connects to. It can do many other things, but just...don't use them?

C) The value is in not having to develop orchestration in house. They already had it so yea, I wouldn't say "throw it out and go to k8s", but if you're starting from scratch and considering between "write and maintain a bunch of bespoke deployment scripts" and "just spin up Talos, write a few yaml files and call it a day" I think the latter is quite compelling.

Re: Modern CI is too complex and misdirected (2021)

#112

Post by maccard »

thechao wrote: post #86
maccard wrote: post #79

Earlier quoted context omitted.

It’s a few million lines of c++ combined with content pipelines. Shader compilation is expensive and the tooling is horrible. Our cached builds on CI are 20 minutes from submit to running on steam which is ok. We also build with MSVC so none of the normal ccache stuff works for us, which is super frustrating

Fuck. I write shader compilers.

Eh, you write them I (ab)use them.

Re: Modern CI is too complex and misdirected (2021)

#113

Post by DrBazza »

nrclark wrote: post #89
DrBazza wrote: post #64

Earlier quoted context omitted.

Your build should be this: build.bash and that's it (and that can even trigger a container build). I've spent far too much time debugging CI builds that work differently to a local build, and it's always because of extra nonsense added to the CI server somehow. I've yet to find a build in my industry that doesn't yield to this 'pattern'. Your environment setup should work equally on a local machine or a CI/CD server,…

Agreed with this sentiment, but with one minor modification: use a Makefile instead. Recipes are still chunks of shell, and they don’t need to produce or consume any files if you want to keep it all task-based. You get tab-completion, parallelism, a DAG, and the ability to start anywhere on the task graph that you want. It’s possible to do all of this with a pure shell script, but then you’re probably reimplementing…

> use a Makefile instead

I was making a general comment that your build should be a single 'command'. Personally, I don't care what the command is, only that it should be a) one command, and b) 100% runnable on a dev box or a server. If you use make, you'll soon end up writing... shell scripts, so just use a shell script.

In an ideal world your topmost command would be a build tool:

     ./gradlew build
     bazel build //...
     make debug
     cmake --workflow --preset
Unfortunately, the second you do that ^^^, someone edits your CI/CD to add a step before the build starts. It's what people do :(

All the cruft that ends up *in CI config*, should be under version control, and inside your single command, so you can debug locally.

Re: Modern CI is too complex and misdirected (2021)

#114

Post by justinrubek »

kqr wrote: post #105
KronisLV wrote: post #33

Earlier quoted context omitted.

> Part of the problem, maybe the whole problem, is that we could get it all working and portable and optimized for non-blessed environments, but it still will only be expected to work over there, and so the frog keeps boiling. Build the software inside of containers (or VMs, I guess): a fresh environment for every build, any caches or previous build artefacts explicitly mounted. Then, have something like this, so tho…

This sounds a lot like "use Nix".

Unfortunately, that's the last thing a lot of people want to hear, despite it saving a whole lot of heartache.

Re: Modern CI is too complex and misdirected (2021)

#115

Post by dapperdrake »

franga2000 wrote: post #46
dapperdrake wrote: post #41

Earlier quoted context omitted.

Unix filesystem inodes and file descriptors stick around until they are closed, even if the inode has been unlinked from a directory. The latter is usually called "deleting the file". All the stuff Erlang does. Static linking and chroot. The problems and the concepts and solutions have been around for a long time. Piles and piles of untold complexity, missing injectivity on data in the name of (leaky) abstractions an…

Ok so let's say you statically link your entire project. There are many reasons you shouldn't or couldn't, but let's say you do. How do you deploy it to the server? Rsync, sure. How do you run it? Let's say a service manager like systemd. Can you start a new instance while the old one is running? Not really, you'll need to add some bash script glue. Then you need a loadbalancer to poll the readiness of the new one an…

Now I see how it could have been confusing to read.

Cannot edit anymore so amending here:

Static liking and chroot (not as The One True Solution (TM)) but as basically Docker without Linux network namespaces.

Linux/Docker actually wound up improving things here. And they got to spend all the money on convincing the people that like advertisements.

And static linking mainly only becomes relevant (and then irrelevant again) in C because if boundaries between compilation units. SQLite throws all of this out. They call it an amalgamation (which also sounds better than a "unity build").

The tools are there. They are just overused. Look at enterprise Hello World in Java for a good laugh.

————

If your data lives in a database on another end if a unix or TCP socket, then I still don't see "NIH". The new binary self-tests and the old binary waits for a shutdown command record and drains its connections.

Kernels and databases clock in at over 5M lines of code. NIH seems like missing the point there.

And most services neither need nor have nine nines of uptime. That is usually too expensive. And always bespoke. Must be tailored to the available hardware.

Code is less portable than people believe.

Ten #ifdef directives and you are often dead on arrival.

Re: Modern CI is too complex and misdirected (2021)

#116

Post by chubot »

DrBazza wrote: post #113
nrclark wrote: post #89

Earlier quoted context omitted.

Agreed with this sentiment, but with one minor modification: use a Makefile instead. Recipes are still chunks of shell, and they don’t need to produce or consume any files if you want to keep it all task-based. You get tab-completion, parallelism, a DAG, and the ability to start anywhere on the task graph that you want. It’s possible to do all of this with a pure shell script, but then you’re probably reimplementing…

> use a Makefile instead I was making a general comment that your build should be a single 'command'. Personally, I don't care what the command is, only that it should be a) one command, and b) 100% runnable on a dev box or a server. If you use make, you'll soon end up writing... shell scripts, so just use a shell script. In an ideal world your topmost command would be a build tool: ./gradlew build bazel build //...…

That's exactly why the "main" should be shell, not make (see my sibling reply). So when someone needs to add that step, it becomes:

    #!/bin/sh

    step-I-added-to-shell-rather-than-CI-yaml
    make debug  # or cmake, bazel
This is better so you can run the whole thing locally, and on different CI providers

In general, a CI is not a DAG, and not completely parallel -- but it often contains DAGs

Re: Modern CI is too complex and misdirected (2021)

#117

Post by DrBazza »

mikepurvis wrote: post #103
DrBazza wrote: post #64

Earlier quoted context omitted.

Your build should be this: build.bash and that's it (and that can even trigger a container build). I've spent far too much time debugging CI builds that work differently to a local build, and it's always because of extra nonsense added to the CI server somehow. I've yet to find a build in my industry that doesn't yield to this 'pattern'. Your environment setup should work equally on a local machine or a CI/CD server,…

There are various things that can be a reasonable candidate for the "top level" build entrypoint, including Nix, bazel, docker bake, and probably more I'm not thinking of. They all have an entrypoint that doesn't have a ton of flags or nonsense, and operate in a pretty self contained environment that they set up and manage themselves. Overall I'm not a fan of wrapping things; if there are flags or options on the top-…

Fortunately most CI/CD systems expose an environment variable during the build so you can detect most of those situations and still write a script that runs locally on a developer box.

Our wrapping is 'minimal', in that you can still run

    bazel build //...
or

    cmake ...
and get the same build artefacts as running:

    build.bash release
My current company is fanatical about read-only for just about every system we have (a bit like Nix, I suppose), and that includes CI/CD. Once the build is defined to run debug or release, rights are removed so the only thing you can edit are the build scripts you have under your control in your repo. This works extremely well for us.

Re: Modern CI is too complex and misdirected (2021)

#118

Post by joelfried »

kstrauser wrote: post #107
captn3m0 wrote: post #90

Earlier quoted context omitted.

They are instead focusing on Agentic Workflows which used natural language instead of YAML. https://github.com/githubnext/gh-aw

Know what I love in a good build system? Nondeterminism! Who needs coffee when you can get your thrills from stochastic processes. Why settle for just non-repeatable builds when you can have non-repeatable build failures!

Would a smart AI accept such foolishness? I doubt it. It'll still use something deterministic under the hood - it'll just have a conversational abstraction layer for talking to the Product person writing up requirements.

We used to have to be able to communicate with other humans to build something. It seems to me that's what they're trying to take out of the loop by doing the things that humans do: talk to other humans and give them what they're asking for.

I too am not a fan of the dystopias we're ending up in.

Re: Modern CI is too complex and misdirected (2021)

#119

Post by reactordev »

MathMonkeyMan wrote: post #15

I remember a Rich Hickey talk where he described Datomic, his database. He said "the problem with a database is that it's over there ." By modeling data with immutable "facts" (a la Prolog), much of the database logic can be moved closer to the application. In his case, with Clojure's data structures. Maybe the the problem with CI is that it's over there . As soon as it stops being something that I could set up and r…

The rule for CI/CD and DevOps in general is boil your entire build process down to one line:

    ./build.sh
If you want to ship containers somewhere, do it in your build script where you check to see if you’re running in “CI”. No fancy pants workflow yamls to vendor lock yourself into whatever CI platform you’re using today, or tomorrow. Just checkout, build w/ params, point your coverage checker at it.

This is also the same for onboarding new hires. They should be able to checkout, and build, no issues or caveats, setup for local environment. This ensures they are ready to PR by end of the day.

(Fmr Director of DevOps for a Fortune 500)

Re: Modern CI is too complex and misdirected (2021)

#120

Post by DrBazza »

chubot wrote: post #104
nrclark wrote: post #89

Earlier quoted context omitted.

Agreed with this sentiment, but with one minor modification: use a Makefile instead. Recipes are still chunks of shell, and they don’t need to produce or consume any files if you want to keep it all task-based. You get tab-completion, parallelism, a DAG, and the ability to start anywhere on the task graph that you want. It’s possible to do all of this with a pure shell script, but then you’re probably reimplementing…

Make is not a general purpose parallel DAG engine. It works well enough for small C projects and similar, but for problems of even medium complexity, it falls down HARD Many years ago, I wrote 3 makefiles from scratch as an exploration of this (and I still use them). I described the issues here: https://lobste.rs/s/yd7mzj/developing_our_position_on_ai#c_s... --- The better style is in a sibling reply -- invoke Make f…

Portability to other CI/CDs systems is an understated reason to use a single build command.
Post reply on HN