Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modern CI is too complex and misdirected (2021)

gregoryszorc.com

Post reply on HN
Print view View original

61–70 of 207 posts

Re: Modern CI is too complex and misdirected (2021)

#61

Post by zokier »

I agree on build systems and CI being closely related, and could (in an ideal world) benefit from far tighter integration. But..

> So here's a thought experiment: if I define a build system in Bazel and then define a server-side Git push hook so the remote server triggers Bazel to build, run tests, and post the results somewhere, is that a CI system? I think it is! A crude one. But I think that qualifies as a CI system.

Yes the composition of hooks, build, and result posting can be thought as a CI system. But then the author goes on to say

> Because build systems are more generic than CI systems (I think a sufficiently advanced build system can do a superset of the things that a sufficiently complex CI system can do)

Which is ignoring the thing that makes CI useful, the continuous part of continuous integration. Build systems are explicitly invoked to do something, CI systems continuosly observe events and trigger actions.

In the conclusion section author mentions this for their idealized system:

> Throw a polished web UI for platform interaction, result reporting, etc on top.

I believe that platform integrations, result management, etc should be pretty central for CI system, and not a side-note that is just thrown on top.

Re: Modern CI is too complex and misdirected (2021)

#62

Post by franga2000 »

s_Hogg wrote: post #54
franga2000 wrote: post #38

Earlier quoted context omitted.

Requirements are complex too. Even if you don't need to scale at all, you likely do need zero-downtime deployment, easy rollbacks, server fault tolerance, service isolation... If you put your apps into containers and throw them onto Kubernetes, you get a lot of that "for free" and in a well-known and well-tested way. Hand-rolling even one of those things, let alone all of them together, would take far too much effort…

Holy shit you don't get anything for _free_ as a result of adopting Kubernetes dude. The cost is in fact quite high in many cases - you adopt Kubernetes and all of the associated idiosyncrasies, which can be a lot more than what you left behind.

For free as in "don't have to do anything to make those features, they're included".

What costs are you talking about? Packaging your app in a container is already quite common so if you already do that all you need to do is replace your existing yaml with a slightly different yaml.

If you don't do that already, it's not really that difficult. Just copy-paste your your install script or rewrite your Ansible playbooks into a Dockerfile. Enjoy the free security boost as well.

What are the other costs? Maintaining something like Talos is actually less work than a normal Linux distro. You already hopefully have a git repo and CI for testing and QA, so adding a "build and push a container" step is a simple one-time change. What am I missing here?

Re: Modern CI is too complex and misdirected (2021)

#64

Post by DrBazza »

MathMonkeyMan wrote: post #15

I remember a Rich Hickey talk where he described Datomic, his database. He said "the problem with a database is that it's over there ." By modeling data with immutable "facts" (a la Prolog), much of the database logic can be moved closer to the application. In his case, with Clojure's data structures. Maybe the the problem with CI is that it's over there . As soon as it stops being something that I could set up and r…

Your build should be this:

    build.bash
and that's it (and that can even trigger a container build).

I've spent far too much time debugging CI builds that work differently to a local build, and it's always because of extra nonsense added to the CI server somehow. I've yet to find a build in my industry that doesn't yield to this 'pattern'.

Your environment setup should work equally on a local machine or a CI/CD server, or your devops teams has identically set it up on bare metal using Ansible or something.

Re: Modern CI is too complex and misdirected (2021)

#65

Post by e1gen-v »

I’ve been using Pulumi automation in our CI and it’s been really nice. There’s definitely a learning curve with the asynchronous Outputs but it’s really nice for building docker containers and separating pieces of my infra that may have different deployment needs.

Re: Modern CI is too complex and misdirected (2021)

#67

Post by homebrewer »

donatj wrote: post #59

Drone was absolutely perfect back when it was Free Software. Literally "run these commands in this docker container on these events" and basically nothing more. We ran the last fully open source version much longer than we probably should have. When they went commercial, GitHub Actions became the obvious choice, but it's just married to so much weirdness and unpredictability. Whole thing with Drone opened my eyes at…

It lives on as Woodpecker, the fork of the last truly free version. As simple as it gets, no CLAs required to contribute.

Re: Modern CI is too complex and misdirected (2021)

#68

Post by homebrewer »

OtherShrezzing wrote: post #48
thrown-0825 wrote: post #4

Earlier quoted context omitted.

until you have to debug a GH action, especially if it only runs on main or is one of the handful of tasks that are only picked up when committed to main. god help you, and don’t even bother with the local emulators / mocks.

What are the good local emulators for gh actions? The #1 reason we don’t use them is because the development loop is appallingly slow.

nektos/act was considered good enough to be adopted as the CI solution for Gitea and Forgejo. The latter uses it for all their development, seems to work out fine for them.

I've never been a fan of GitHub Actions (too locked-in/proprietary for my taste), so no idea if it lives up to expectations.

Re: Modern CI is too complex and misdirected (2021)

#69

Post by zahlman »

taminka wrote: post #39
mettamage wrote: post #34

IMO development is too complex and misdirected in general since we cargo cult FAANG. Need AWS, Azure or GCP deployment? Ever thought about putting it on bare metal yourself? If not, why not? Because it's not best practice? Nonsense. The answer with these things is: it depends, and if your app has not that many users, you can get away with it, especially if it's a B2B or internal app. It's also too US centric. The ide…

many ppl also underestimate how capable modern hardware is: for ~10usd you could handle like a million concurrent connections with a redis cluster on a handful of VPSs...

Relevant: Program Your Own Computer in Python (https://www.youtube.com/watch?v=ucWdfZoxsYo) from this year's PyCon, emphasizing how much you can accomplish with local execution and how much overhead can be involved in doing it remotely.

Re: Modern CI is too complex and misdirected (2021)

#70

Post by maccard »

MortyWaves wrote: post #51
MathMonkeyMan wrote: post #15

I remember a Rich Hickey talk where he described Datomic, his database. He said "the problem with a database is that it's over there ." By modeling data with immutable "facts" (a la Prolog), much of the database logic can be moved closer to the application. In his case, with Clojure's data structures. Maybe the the problem with CI is that it's over there . As soon as it stops being something that I could set up and r…

It’s why I’ve started making CI simply a script that I can run locally or on GitHub Actions etc. Then the CI just becomes a bit of yaml that runs my script.

How does that script handle pushing to ghcr, or pulling an artifact from a previous stage for testing?

In my experience these are the bits that fail all the time, and are the most important parts of CI once you go beyond it taking 20/30 seconds to build.

A clean build in an ephemeral VM of my project would take about 6 hours on a 16 core machine with 64GB RAM.

Post reply on HN