Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The CD Pipeline Manifesto

manifesto.getglu.dev

Post reply on HN
Print view View original

31–40 of 51 posts

Re: The CD Pipeline Manifesto

#31

Post by ttyprintk »

clvx wrote: post #10

I have a hot take on this. I don’t care how you build and deploy as long as it’s reproducible and the whole process can be tracked in their metadata. I’d rather have a process validating CI/CD stages and artifacts metadata in a central db than unifying pipelines that won’t get standardized due communication complexity. This way I can have a conversation on visibility rather than code edge cases.

This is important for SBOM (software bill-of-materials) which will soon be mandatory in regulated domains.

Re: The CD Pipeline Manifesto

#32

Post by vergessenmir »

Is any of this reproducible? Not sure why that requirement has been quietly overlooked.

I've worked in this space for a long time and can't make head or tail of what glue is.

A motivating examplt would be help which I might have missed?

Re: The CD Pipeline Manifesto

#33

Post by maccard »

pdimitar wrote: post #29
drowsspa wrote: post #28

Earlier quoted context omitted.

There are a pletora of tools better than Make. But it's a standard tool, everyone is familiar with it, you probably don't even have to install it.

`just` is 90% similar to `make` in syntax, only it has 100x less foot guns. :) Also I'll never understand the appeal of "not having to install a tool". We're not in the 1980s anymore when that was an actual chore. You run a command, the tool is there (including in CI/CD), boom, done. What am I missing here? The advantages you list are flimsy at best.

Bootstrapping can be painful in some languages or frameworks. Not everyone is running containerised builds where there are ephemeral environments that you just install a tool (and pay the 30+ second cost per build to run apt-get update). There’s certainly value in having a front door entry point. But I think it should be a shell script, not a makefile.

Re: The CD Pipeline Manifesto

#34

Post by pdimitar »

maccard wrote: post #33
pdimitar wrote: post #29

Earlier quoted context omitted.

`just` is 90% similar to `make` in syntax, only it has 100x less foot guns. :) Also I'll never understand the appeal of "not having to install a tool". We're not in the 1980s anymore when that was an actual chore. You run a command, the tool is there (including in CI/CD), boom, done. What am I missing here? The advantages you list are flimsy at best.

Bootstrapping can be painful in some languages or frameworks. Not everyone is running containerised builds where there are ephemeral environments that you just install a tool (and pay the 30+ second cost per build to run apt-get update). There’s certainly value in having a front door entry point. But I think it should be a shell script, not a makefile.

Yes to your last. Either sh/bash script or a precompiled Golang program. If installing a tool is really such a problem then having a precompiled strongly typed program doing various tasks should be a no-brainer.

I started openly hating `make` because I re-learned its specifics and quirks several times over the course of 10-ish years and then figured that I want to learn stuff with a staying power in my brain. I don't use `make` every work day so eventually any quirks disappear -- that's how our brains work.

So that's why I learned most of `just` and it hasn't betrayed me so far, not once. Though I did write a few Elixir and Golang programs for running various tasks in production environment, too.

Re: The CD Pipeline Manifesto

#35

Post by sourishkrout »

GrumpyCat42 wrote: post #17
sourishkrout wrote: post #7

I commend anyone who’s taking a hard look at our current CI/CD practices. Good work! Succinctly stating the problems is easier said than done. I believe https://dagger.io checks all these manifesto boxes and more. At least that’s where I’m focusing my attention.

My company is currently adopting this and I don't see the appeal yet - likely from a lack of knowing much about it. I added it to a side project just to get familiar and it added quite a few sdk files and folders to my project, and lots of decorators. It also required Docker and yadda yadda yadda. I just could not justify using it compared to just running some regular Typescript file with Bun (or, in a different proj…

Simple problems require simple solutions. If Makefile, NPM run, or Rake gets the job done, stick with it. That's great.

The problem that Dagger and similar efforts solve is for pipelines at scale, whether that's a sea of microservices maintained by an armada of teams (which never work the same) or your massive pipelines that should be decomposed into a more atomic pipeline that fed into one.

I believe the latter is a big productivity hurdle even without org-scale. My release pipeline runs for 25min with a team However, god forbid it fails with a non-obvious error of 20 minutes into exec. Lack of portability (Hi GHA vendor lock-in) and reproducibility (local-run = impossible) will make this feedback-loop hell.

Now, wiseguys might tell me that pipelines shouldn't run for multiple minutes and only unit tests blah. That's divorced from reality. This sentiment won't not solve automation problems and won't optimize for velocity. It merely throws it over the fence to somebody else. If you have "the luxury" a QE/QA/Release team which I feel bad for.

So the question to ask yourself is: how do I know I have outgrown `go run cmd/ci/main.go`?

Re: The CD Pipeline Manifesto

#36

Post by eddsolves »

ttyprintk wrote: post #31
clvx wrote: post #10

I have a hot take on this. I don’t care how you build and deploy as long as it’s reproducible and the whole process can be tracked in their metadata. I’d rather have a process validating CI/CD stages and artifacts metadata in a central db than unifying pipelines that won’t get standardized due communication complexity. This way I can have a conversation on visibility rather than code edge cases.

This is important for SBOM (software bill-of-materials) which will soon be mandatory in regulated domains.

What will SBOM require in regulated domains?

Re: The CD Pipeline Manifesto

#37

Post by tom_ »

maccard wrote: post #16
jas39 wrote: post #11

I'm thinking before we build a CI/CI pipeline, make sure there is a Makefile. Why have they gone out of style?

If I want to build and test a golang app and push it to a container repository, what value does a makefile provide over go build && docker push? All the tools do their own dependency tracking already (unfortunately).

You want to use GNU Make, and then you can ignore the Make dependency tracking. GNU Make is much easier when you only use so-called phony targets (consult the manual), which always execute without doing any dependency tracking.

As for the advantage, a makefile will definitely perform both go build and docker push, rather than just (say) docker push, an ever-present risk if you have to rely on your fingers to type these things in, or rely on your eyes to check that you recalled the right command from the history. It will also explicitly tell you the build failed rather than relying on you to do echo $? or for the tools to have some obvious output in the error case.

A shell script is also an option. Makefiles have some helpful extra features: by default, commands are echoed; by default, the build fails if any command exits with a non-0 exit code (please consult local Unix person for the details); a Makefile inherently has multiple entry points; and, a Makefile can also be easier to get working on Windows than a shell script, though if you can demand that people run it from Git Bash then maybe there's not much in this.

If you're still not convinced: that's fine! This is not a sales pitch.

(I've more recently switching to using a do-everything Python script for this stuff, which is more annoying when it comes to invoking commands but has obvious advantages in terms of how easy it is to add extra logic and checks and some UI creature comforts and so on.)

Re: The CD Pipeline Manifesto

#38

Post by a1o »

I don't think this is reasonable if you have a cross platform (web, android, iOS, macOS, Windows, Linux, FreeBSD, ...) app, things won't be that clean, that only works if whatever you do is very simple, otherwise there will be some patchwork to build and test across all platforms - there's just no way to run all of them local in your single platform computer whatever that is. Honestly a lot of what is there is not that useful, I don't need types in the pipeline when scripting python for build integration logic, there's nothing that types brings that are a must in that case.

Re: The CD Pipeline Manifesto

#39

Post by a1o »

ttyprintk wrote: post #30
GrumpyCat42 wrote: post #18

Earlier quoted context omitted.

My struggle with Make and bash is that they're not very expressive - maybe that's something we want in our CIs, but I've always preferred writing an actual program in that program's native language for CI/CD, even if it has to shell out some commands every now and again.

I prefer that, too. I've heard (less-experienced) tech leads forbid Makefiles because they're not declarative enough compared to yaml.

If you stick with what is in common between ninja build and Makefiles, and comment any usage you do of what isn't, the file will mostly mention a series of inputs -> box -> outputs. What happens is make will dispatch in a way that the inputs are all satisfied. It works fine afaict, the only issue is make doesn't contain by itself the tools that it expects to be available in your environment, so you will still need something else to solve that.

Re: The CD Pipeline Manifesto

#40

Post by tom_ »

drowsspa wrote: post #27
maccard wrote: post #26

Earlier quoted context omitted.

At that point you end up with a makefile that has a 1:1 mapping with targets in my experience. At a previous job, we had an enormous makefile, most of which was defining phone targets and translating make arguments into maven arguments. All the actual targets were calling maven. Make provided no value at all in that, other than requiring you to know Make and maven to modify anything in the build. Personally I’d rathe…

Yeah, mine often just defers back to some shell scripts. But it's useful to enumerate them in the Makefile.

If you have a telltale prefix for any internal phony targets (I use "_"), then you can have the Makefile list all the interesting targets itself. Cat the Makefile, print every line matching "^\.PHONY:[[:space:]]*[^_]", then strip out the prefix. Leave any suffix, as you can put a trailing comment in, e.g.,

    .PHONY:build_windows # build for Windows. Supply DEBUG=1 for a debug build
I find this super useful. Even if you remember exactly what the target was called, it still gives you a nice list of words you can double click on to pull into the command line.
Post reply on HN