Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modern CI is too complex and misdirected

gregoryszorc.com

Post reply on HN
Print view View original

161–170 of 184 posts

Re: Modern CI is too complex and misdirected

#161

Post by kvnhn »

> In my ideal world, there is a single DAG dictating all build, testing, and release tasks.

I've stewed on this as well, and I'll add my two cents:

I'd argue this problem -- building and managing a DAG of tasks -- isn't just critical to build systems and CI. It's everywhere: cloud architecture, Dev/MLOps pipelines... I'd argue most programming is just building and managing a DAG of tasks (e.g. functions, and purposefully limiting to "acyclic" for this argument). This is why we always regress to Turing complete languages; they're great at building DAGs.

So yes, I agree that some standard DAG language (probably Turing complete) would be great. But I'd extend it's reach into source code itself. Pass your DAG to Python, and it schedules and runs your tasks in the interpreter (or perhaps many interpreters). Pass your DAG to Kubernetes and it schedules and runs your tasks in containers. etc.

inb4 Lisp

Re: Modern CI is too complex and misdirected

#162

Post by [deleted] »

tmh88j wrote: post #130
cosmodisk wrote: post #116

Earlier quoted context omitted.

I have someone in my network, who is very active in PHP scene. Tutorials, tips&tricks, code reviews, you name it. He pretty much abandoned his very popular website and went all in on YouTube. Why? Apparently watching video is so much easier than reading 3000 word article.

>He pretty much abandoned his very popular website and went all in on YouTube. Why? Apparently watching video is so much easier than reading 3000 word article. Learning from videos is lazy? Content you've learned is only valid if you read it? I'm not sure what exactly you're getting at, but I'm a visual learner and I much prefer (well made) videos over text. There's a visual and audio aspect to it enabling so much mo…

[deleted]

Re: Modern CI is too complex and misdirected

#163

Post by HWR_14 »

noir_lord wrote: post #24
jrockway wrote: post #17

I think that modern CI is actually too simple. They all boil down to "get me a Linux box and run a shell script". You can do anything with that, and there are a million different ways to do everything you could possibly want. But, it's easy to implement, and every feature request can be answered with "oh, well just apt-get install foobarbaz3 and run quuxblob to do that." A "too complex" system, would deeply integrate…

Pretty much. Docker in my experience is the same way, people see docker as the new hotness then treat it like a Linux box with a shell script (though at least with the benefit you can shoot it in the head). One of the other teams had an issue with reproducibility on something they where doing so I suggested that they use a multistage build in docker and export the result out as an artefact they could deploy, they loo…

>Also feels like people adopt tools and cobble shit together from google/SO, what happened to RTFM.

I find that documentation has gotten worse and harder to find. It used to be trivial to find and long to read. Now, between SO and SEO of pages designed to bring people in, it's 95% examples not enough "here's all the data about how x works". It's very easy to get a thing going, but very hard to understand all the things that can be done.

Re: Modern CI is too complex and misdirected

#164

Post by HWR_14 »

tmh88j wrote: post #130
cosmodisk wrote: post #116

Earlier quoted context omitted.

I have someone in my network, who is very active in PHP scene. Tutorials, tips&tricks, code reviews, you name it. He pretty much abandoned his very popular website and went all in on YouTube. Why? Apparently watching video is so much easier than reading 3000 word article.

>He pretty much abandoned his very popular website and went all in on YouTube. Why? Apparently watching video is so much easier than reading 3000 word article. Learning from videos is lazy? Content you've learned is only valid if you read it? I'm not sure what exactly you're getting at, but I'm a visual learner and I much prefer (well made) videos over text. There's a visual and audio aspect to it enabling so much mo…

It depends on what you're learning. Try to learn to chop veggies and a video is superior to text. Try to learn all the inputs to a function, return values and thrown errors and a video is far inferior to text. The crossover point probably varies from person to person, and my guess is snobbery about videos is based around the personal differences in how a person learns and where what they already know or what they learn is on that spectrum.

There's also a wide (wider than text?) difference in video quality. How many videos out there are badly narrated sources available in text format?

Re: Modern CI is too complex and misdirected

#165

Post by chubot »

jcranmer wrote: post #86
pydry wrote: post #11

It's weird that people keep building DSLs or YAML based languages for build systems. It's not a new thing, either - I remember using whoops-we-made-it-turing complete ANT XML many years ago. Build systems inevitably evolve into something turing complete. It makes much more sense to implement build functionality as a library or set of libraries and piggyback off a well designed scripting language.

The way I would categorize build systems (and by extension, a lot of CI systems) is semi-declarative. That is to say, we can describe the steps needed to build as a declarative list of source files, the binaries they end up in, along with some special overrides (maybe this one file needs special compiler flags) and custom actions (including the need to generate files). To some degree, it's recursive: we need to build…

I think what you are getting at is a "staged execution model", and I agree.

GNU make actually has this, but it's done poorly. It has build STEPS in the shell language, but the build GRAPH can be done in the Make language, or even Guile scheme. [1]

----

I hope to add the "missing declarative part" to shell with https://www.oilshell.org.

So the build GRAPH should be described as you say. It's declarative, but you need metaprogramming. You can think of it like generating a Ninja file, but using reflection/metaprogramming rather than textual code generation.

And then the build STEPS are literally shell. Shell is a lot better than Python for this use case! e.g. for invoking cmopilers and other tools.

I hinted at this a bit in a previous thread: https://news.ycombinator.com/item?id=25343716

And this current comment https://lobste.rs/s/k0qhfw/modern_ci_is_too_complex_misdirec...

Comments welcome!

[1] aside: Tensorflow has the same staged execution model. The (serial) Python language is used for metaprogramming the graph, while the the highly parallel graph language is called "Tensorflow".

Re: Modern CI is too complex and misdirected

#166

Post by stingraycharles »

gryn wrote: post #117
stingraycharles wrote: post #31

Earlier quoted context omitted.

I agree with you in principle, but I have learned to accept that this only works for 80% of the functionality. Maybe this works for a simple Diablo or NodeJS project, but in any large production system there is a gray area of “messy shit” you need, and a CI system being able to cater to these problems is a good thing. Dockerizing things is a step in the right direction, at least from the perspective of reproducibilit…

I'm curious what's a Diablo Project ? I've never heard of such technology unless you're speaking of the game with the same name. Did you possibly mean Django ?

Argh it was indeed Django, I was on mobile and it must have been autocorrected.

Re: Modern CI is too complex and misdirected

#167

Post by Mic92 »

cyberpunk wrote: post #122
ElijahLynn wrote: post #114

Earlier quoted context omitted.

> GitHub Actions, et. al. cannot touch the target environments that we deploy our software to. It can with on-prem self-hosted Runners > https://docs.github.com/en/actions/hosting-your-own-runners/... I just had this same complaint about using Actions and was pointed to this document.

Yeah and that's a total shitshow too. You want to run that in k8s? * First off it won't work with a musl image * You need to request a new token every 2 hours * It doesn't spawn pods per build like gitlab, it's literally a dumb runner, the jobs will execute *IN* the runner container, so no isolation, and you need all the tools under the sun installed in the container (our runner image clocked in at 2gb for a java/nod…

Yeah. I really hope they fix the dumb runner part: https://github.com/actions/runner/pull/660 I resorted to this one instead https://github.com/summerwind/actions-runner-controller but it requires kubernetes...

Re: Modern CI is too complex and misdirected

#168

Post by oblio »

mumblemumble wrote: post #78
oblio wrote: post #62

Earlier quoted context omitted.

People hate on it, but do you know what language would be perfect these days? Easy shelling - check. Easily embeddable - check Easily sandboxable - check. Reasonably rich standard library - check. High level abstractions - check. If you're still guessing what language it is, it's Tcl. Good old Tcl. It's just that is syntax is moderately weird and the documentation available for it is so ancient and creaky that you ca…

How is the Windows support? One of my big needs for any general-purpose build system is that I can get a single build that works on both Windows and POSIX. Without using WSL. That said, you're right, at least at first blush, tcl is an attractive, if easy to forget, option.

Windows support? Great as far as I can tell. ActiveTcl, the Windows distribution, has been a thing for several decades now. I remember using it back in 2008.

Re: Modern CI is too complex and misdirected

#169

Post by oblio »

lstamour wrote: post #50
qznc wrote: post #36

Earlier quoted context omitted.

How does Bazel deal with different platforms? For example, run tests on Windows, BSD, Android, Raspberry Pi, RISCv5, or even custom hardware?

The reason this isn't a concern is because Bazel tries very hard to not let any system libraries or configurations interfere with the build, at all, ever. So it should rarely matter what platform you're running a build on, the goal should be the same output every time from every platform. Linux is recommended, or a system that can run Docker and thus Linux. From there it depends on the test or build step. I haven't d…

> That said Bazel is portable - it generally needs Java and Bash and is generally portable to platforms that have both, though I haven't checked recently. There are exceptions though, and it will run natively in Windows, just not as easily. https://docs.bazel.build/versions/master/windows.html It also works on Mac, but it's missing Linux disk sandboxing features and makes up for it using weird paths and so on.

The good old: in theory it's portable, but in practice the target of that port better look 100% like Linux :-)

Re: Modern CI is too complex and misdirected

#170

Post by oblio »

0xbadcafebee wrote: post #144
lenkite wrote: post #82

I so oppressed with YAML chosen as the configuration language for mainstream CI systems. How do people manage to live with this ? I always make mistakes - again and again. And I can never keep anything in my head. It's just not natural. Why couldn't they choose a programming language ? Restrict what can be done by all means, but something that has a proper parser, compiler errors and IDE/editor hinting support would…

YAML is not a language, it's a data format. Why does nobody in the entire tech industry know the difference? I didn't even go to school and I figured it out. Most software today that uses YAML for a configuration file is taking a data format (YAML) applying a very shitty parser to create a data structure, and then feeding that data structure to a function, which then determines what other functions to call. There's n…

> YAML is not a language, it's a data format.

Yet Another Markup > (which later supposedly became "YAML Ain't Markup Language", because every villain needs a better backstory).

Post reply on HN