Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modern CI is too complex and misdirected (2021)

gregoryszorc.com

Post reply on HN
Print view View original

41–50 of 207 posts

Re: Modern CI is too complex and misdirected (2021)

#41

Post by dapperdrake »

franga2000 wrote: post #38
mettamage wrote: post #34

IMO development is too complex and misdirected in general since we cargo cult FAANG. Need AWS, Azure or GCP deployment? Ever thought about putting it on bare metal yourself? If not, why not? Because it's not best practice? Nonsense. The answer with these things is: it depends, and if your app has not that many users, you can get away with it, especially if it's a B2B or internal app. It's also too US centric. The ide…

Requirements are complex too. Even if you don't need to scale at all, you likely do need zero-downtime deployment, easy rollbacks, server fault tolerance, service isolation... If you put your apps into containers and throw them onto Kubernetes, you get a lot of that "for free" and in a well-known and well-tested way. Hand-rolling even one of those things, let alone all of them together, would take far too much effort…

Unix filesystem inodes and file descriptors stick around until they are closed, even if the inode has been unlinked from a directory. The latter is usually called "deleting the file".

All the stuff Erlang does.

Static linking and chroot.

The problems and the concepts and solutions have been around for a long time.

Piles and piles of untold complexity, missing injectivity on data in the name of (leaky) abstractions and cargo-culting have been with us on the human side if things for even longer.

And as always: technical and social problems may not always benefit from the same solutions.

Re: Modern CI is too complex and misdirected (2021)

#42

Post by dapperdrake »

taminka wrote: post #39
mettamage wrote: post #34

IMO development is too complex and misdirected in general since we cargo cult FAANG. Need AWS, Azure or GCP deployment? Ever thought about putting it on bare metal yourself? If not, why not? Because it's not best practice? Nonsense. The answer with these things is: it depends, and if your app has not that many users, you can get away with it, especially if it's a B2B or internal app. It's also too US centric. The ide…

many ppl also underestimate how capable modern hardware is: for ~10usd you could handle like a million concurrent connections with a redis cluster on a handful of VPSs...

This

Re: Modern CI is too complex and misdirected (2021)

#43

Post by karel-3d »

jph wrote: post #2

You're 100% right IMHO about the convergence of powerful CI pipelines and full build systems. I'm very curious what you'll think if you try Dagger, which is my tool of choice for programming the convergence of CI and build systems. (Not affiliated, just a happy customer) https://dagger.io/

I absolutely don't understand what it does from the website. (And there is way too much focus on "agents" on the front page for my tastes, but I guess it's 2025)

edit: all the docs are about "agents"; I don't want AI agents, is this for me at all?

Re: Modern CI is too complex and misdirected (2021)

#44

Post by dapperdrake »

MathMonkeyMan wrote: post #15

I remember a Rich Hickey talk where he described Datomic, his database. He said "the problem with a database is that it's over there ." By modeling data with immutable "facts" (a la Prolog), much of the database logic can be moved closer to the application. In his case, with Clojure's data structures. Maybe the the problem with CI is that it's over there . As soon as it stops being something that I could set up and r…

Transactions and a single consistent source of truth with stuff like observability and temporal ordering is centralized and therefore "over there" for almost every place you could be in.

As long as communications have bounded speed (speed of light or whatever else) there will be event horizons.

The point of a database is to track changes and therefore time centrally. Not because we want to, but because everything else has failed miserably. Even conflicting CRDT change merges and git merges can get really hairy really quickly.

People reinvent databases about every 10 years. Hardware gets faster. Just enjoy the show.

Re: Modern CI is too complex and misdirected (2021)

#45

Post by reactordev »

taminka wrote: post #39
mettamage wrote: post #34

IMO development is too complex and misdirected in general since we cargo cult FAANG. Need AWS, Azure or GCP deployment? Ever thought about putting it on bare metal yourself? If not, why not? Because it's not best practice? Nonsense. The answer with these things is: it depends, and if your app has not that many users, you can get away with it, especially if it's a B2B or internal app. It's also too US centric. The ide…

many ppl also underestimate how capable modern hardware is: for ~10usd you could handle like a million concurrent connections with a redis cluster on a handful of VPSs...

One beelink in a closet runs our entire OP’s cluster.

Re: Modern CI is too complex and misdirected (2021)

#46

Post by franga2000 »

dapperdrake wrote: post #41
franga2000 wrote: post #38

Earlier quoted context omitted.

Requirements are complex too. Even if you don't need to scale at all, you likely do need zero-downtime deployment, easy rollbacks, server fault tolerance, service isolation... If you put your apps into containers and throw them onto Kubernetes, you get a lot of that "for free" and in a well-known and well-tested way. Hand-rolling even one of those things, let alone all of them together, would take far too much effort…

Unix filesystem inodes and file descriptors stick around until they are closed, even if the inode has been unlinked from a directory. The latter is usually called "deleting the file". All the stuff Erlang does. Static linking and chroot. The problems and the concepts and solutions have been around for a long time. Piles and piles of untold complexity, missing injectivity on data in the name of (leaky) abstractions an…

Ok so let's say you statically link your entire project. There are many reasons you shouldn't or couldn't, but let's say you do. How do you deploy it to the server? Rsync, sure. How do you run it? Let's say a service manager like systemd. Can you start a new instance while the old one is running? Not really, you'll need to add some bash script glue. Then you need a loadbalancer to poll the readiness of the new one and shift the load. What if the new instance doesn't work right? You need to watch for that, presumably with another bash script, stop it and keep the old one as "primary". Also, you'll need to write some selinux rules to make it so if someone exploits one service, they can't pivot to others.

Congrats, you've just rewritten half of kubernetes in bash. This isn't reducing complexity, it's NIH syndrome. You've recreated it, but in a way that nobody else can understand or maintain.

Re: Modern CI is too complex and misdirected (2021)

#48

Post by OtherShrezzing »

thrown-0825 wrote: post #4
tacker2000 wrote: post #3

These online / paid CI systems are a dime a dozen and who knows what will happen to them in the future… Im still rocking my good old jenkins machine, which to be fair took me a long time to set up, but has been rock solid ever since and will never cost me much and will never be shut down. But i can definitely see the appeal of github actions, etc…

until you have to debug a GH action, especially if it only runs on main or is one of the handful of tasks that are only picked up when committed to main. god help you, and don’t even bother with the local emulators / mocks.

What are the good local emulators for gh actions? The #1 reason we don’t use them is because the development loop is appallingly slow.

Re: Modern CI is too complex and misdirected (2021)

#49

Post by GuB-42 »

qwertytyyuu wrote: post #6

Wait a CI isn't supposed to be a build system that also runs tests?

A CI system is more like a scheduler.

To make things simple: make is a build system, running make in a cron task is CI.

There is nothing special about tests, it is just a step in the build process that you may or may not have.

Re: Modern CI is too complex and misdirected (2021)

#50

Post by sambuccid »

I'm not sure why no one mentioned it yet, but the CI tool of sourcehut (https://man.sr.ht/builds.sr.ht/) simplifies all of this. It just spins a linux distro of your choice, and executes a very bare bone yml that essentially contains a lot of shell commands, so it's also easy to replicate locally.

There are 12 yml keywords in total that cover everything.

Other cool things are the ability to ssh in a build if it failed(for debugging), and to run a one-time build with a custom yml without committing it(for testing).

I believe it can checkout any repository, not just one in sourcehut that triggers a build, and that has also a GraphQL API

Post reply on HN