Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modern CI is too complex and misdirected (2021)

gregoryszorc.com

Post reply on HN
Print view View original

141–150 of 207 posts

Re: Modern CI is too complex and misdirected (2021)

#141

Post by merb »

taminka wrote: post #39
mettamage wrote: post #34

IMO development is too complex and misdirected in general since we cargo cult FAANG. Need AWS, Azure or GCP deployment? Ever thought about putting it on bare metal yourself? If not, why not? Because it's not best practice? Nonsense. The answer with these things is: it depends, and if your app has not that many users, you can get away with it, especially if it's a B2B or internal app. It's also too US centric. The ide…

many ppl also underestimate how capable modern hardware is: for ~10usd you could handle like a million concurrent connections with a redis cluster on a handful of VPSs...

many ppl also understimate how complex it is to satisfy uptime requirements, how to scale out local infrastructure when storage > 10/50/100tb (yeah a single disk can handle that, but what about bit rot, raid stuff, etc) is involved.

it gets worse when you need more servers because your ocr process of course needs cpu x so on a beefiy machine you can handle maybe 50 high page documents. but how do you talk to other machines, etc.

also humans costs way more money than cloud stuff. I the cloud stuff can be managed in like 1 day per month you dont need a real person, if you have real hardware that day is not enough and you soon need a dedicated person, keeping everything up-to-date, etc.

Re: Modern CI is too complex and misdirected (2021)

#142

Post by arunix »

bob1029 wrote: post #32

I've been able to effectively skip the entire CI/CD conversation by preferring modern .NET and SQLite. I recently spent a day trying to get a GH Actions build going but got frustrated and just wrote my own console app to do it. Polling git, tracking a commit hash and running dotnet build is not rocket science. Putting this agent on the actual deployment target skips about 3 boss fights.

Is there something about .NET that makes this easier?

Re: Modern CI is too complex and misdirected (2021)

#143

Post by dehrmann »

kstrauser wrote: post #107
captn3m0 wrote: post #90

Earlier quoted context omitted.

They are instead focusing on Agentic Workflows which used natural language instead of YAML. https://github.com/githubnext/gh-aw

Know what I love in a good build system? Nondeterminism! Who needs coffee when you can get your thrills from stochastic processes. Why settle for just non-repeatable builds when you can have non-repeatable build failures!

What I'm hearing is we need to invent LLM-based compilers.

Re: Modern CI is too complex and misdirected (2021)

#144

Post by nlawalker »

>CI offerings like GitHub Actions and GitLab Pipelines are more products than platforms because they tightly couple an opinionated configuration mechanism (YAML files) and web UI (and corresponding APIs) on top of a theoretically generic remote execute as a service offering. For me to consider these offerings as platforms, they need to grow the ability to schedule arbitrary compute via an API, without being constrained by the YAML officially supported out of the box.

I wish the author gave more concrete examples about what kinds of workflows they want to dynamically construct and remotely execute (and why a separate step of registering the workflow up front with the service before running it is such a dealbreaker), and what a sufficiently generic and unopinionated definition schema for workflows and tasks would look like as opposed to what a service like GitHub Actions defines.

Generally, registering a workflow with the service (putting it in your repo, in the case of GHA) makes sense because you're running the same workflows over and over. In terms of task definitions, GHA is workflows -> jobs -> tasks -> actions, where jobs are tied to runners and can have dependencies defined between them. If you want to use those primitives to do something generic like run some scripts, you can do that in a very bare-bones way. When I look at the Taskcluster task definition they linked, I see pretty much the same thing.

Re: Modern CI is too complex and misdirected (2021)

#145

Post by mrbombastic »

reactordev wrote: post #119
MathMonkeyMan wrote: post #15

I remember a Rich Hickey talk where he described Datomic, his database. He said "the problem with a database is that it's over there ." By modeling data with immutable "facts" (a la Prolog), much of the database logic can be moved closer to the application. In his case, with Clojure's data structures. Maybe the the problem with CI is that it's over there . As soon as it stops being something that I could set up and r…

The rule for CI/CD and DevOps in general is boil your entire build process down to one line: ./build.sh If you want to ship containers somewhere, do it in your build script where you check to see if you’re running in “CI”. No fancy pants workflow yamls to vendor lock yourself into whatever CI platform you’re using today, or tomorrow. Just checkout, build w/ params, point your coverage checker at it. This is also the…

How do you get caching of build steps with this approach? Or do you just not?

Re: Modern CI is too complex and misdirected (2021)

#146

Post by kstrauser »

dehrmann wrote: post #143
kstrauser wrote: post #107

Earlier quoted context omitted.

Know what I love in a good build system? Nondeterminism! Who needs coffee when you can get your thrills from stochastic processes. Why settle for just non-repeatable builds when you can have non-repeatable build failures!

What I'm hearing is we need to invent LLM-based compilers.

Time to launch LLMLLVM.

Re: Modern CI is too complex and misdirected (2021)

#147

Post by bluGill »

thechao wrote: post #75
maccard wrote: post #70

Earlier quoted context omitted.

How does that script handle pushing to ghcr, or pulling an artifact from a previous stage for testing? In my experience these are the bits that fail all the time, and are the most important parts of CI once you go beyond it taking 20/30 seconds to build. A clean build in an ephemeral VM of my project would take about 6 hours on a 16 core machine with 64GB RAM.

Sheesh. I've got a multimillion line modern C++ protect that consists of a large number of dylibs and a few hundred delivered apps. A completely cache-free build is an only few minutes. Incremental and clean (cached) builds are seconds, or hundreds of milliseconds. It sounds like you've got hundreds of millions of lines of code! (Maybe a billion!?) How do you manage that?

I have 15 million lines of C++, and builds are several hours. We split into multi-repo (for other reasons) and that helps because compiling is memory bandwidth limited - on the CI system by we can split the different repos to different CI nodes.

Re: Modern CI is too complex and misdirected (2021)

#149

Post by arianvanp »

mrbombastic wrote: post #145
reactordev wrote: post #119

Earlier quoted context omitted.

The rule for CI/CD and DevOps in general is boil your entire build process down to one line: ./build.sh If you want to ship containers somewhere, do it in your build script where you check to see if you’re running in “CI”. No fancy pants workflow yamls to vendor lock yourself into whatever CI platform you’re using today, or tomorrow. Just checkout, build w/ params, point your coverage checker at it. This is also the…

How do you get caching of build steps with this approach? Or do you just not?

Use a modern hermetic build system with remote caching or remote execution. Nix, Bazel, buck, pants. Many options

Re: Modern CI is too complex and misdirected (2021)

#150

Post by jcranmer »

nlawalker wrote: post #144

> CI offerings like GitHub Actions and GitLab Pipelines are more products than platforms because they tightly couple an opinionated configuration mechanism (YAML files) and web UI (and corresponding APIs) on top of a theoretically generic remote execute as a service offering. For me to consider these offerings as platforms, they need to grow the ability to schedule arbitrary compute via an API, without being constrai…

> I wish the author gave more concrete examples about what kinds of workflows they want to dynamically construct and remotely execute (and why a separate step of registering the workflow up front with the service before running it is such a dealbreaker), and what a sufficiently generic and unopinionated definition schema for workflows and tasks would look like as opposed to what a service like GitHub Actions defines.

Something that comes up for me a lot at my work: running custom slices of the test suite. The full test suite probably takes CPU-days to run, and if I'm only interested in the results of something that takes 5 CPU-minutes to run, then I shouldn't have to run all the tests.

Post reply on HN