Earlier quoted context omitted.
This goes against every incentive for the CI service provider
Not necessarily. For example, Buildkite lets you host your own runners.
[deleted]
I hate GitHub Actions with passion
301–310 of 347 posts
Re: I hate GitHub Actions with passion
#302lol, this post makes me feel like crying at my job in BigTech co. He’s complaining about a 30 min feedback loop maybe, mine is 2+ hours.
I write a piece of code, however small the change. Then run the proc, first it takes ~20 mins to compile. Then since it is ML, it can only run on a remote server. It takes an easy 20 more minutes to start running in the remote server. Then after another 40 minutes I can confirm the job failed. The only way to debug is to read through a massive log file, for which we have an in built log reader lol. The log file will have thousands of errors that literally don’t matter, and you’ll never have enough context to know which errors don’t matter. You can simply ping someone and ask, does this error matter, could this be the reason my entire proc failed, oh no this is just a useless log that fails all the time, I shouldn’t have been wasting a day digging into this, ok thanks bye.
But this isn’t it, not even close lol, we in fact have a custom DSL to define computational graphs, which of course does not have any linter, or even any compiler and a very broken visualizer but our entire org runs on this. Syntax errors, logic errors, actual race conditions are all caught the exact same way —- as the process dying after trying to run on a remote server for 2+ hours with no useful error log. So our workflow is to just get a cup of coffee and stare at the graph which can get thousands and thousands of lines big to find at times completely trivial bugs that any half decent language would have caught as a syntax error.
My exp in BigTechCo makes me completely understand GitHub actions, my guess is many big companies have equally janky tools that harm dev productivity but still somehow take absolutely massive workloads. GitHub just thought of sharing it to the public.
Re: I hate GitHub Actions with passion
#303Earlier quoted context omitted.
I find that shell scripting has a sharp cliff. I agree with the sentiment that most things are over engineered. However it’s really easy to go from a simple shell script running a few commands to something significantly more complex just to do something seemingly simple, like parse a semantic version, make an api call and check the status code etc, etc. The other problem with shell scripting on things like GHA is tha…
> However it’s really easy to go from a simple shell script running a few commands to something significantly more complex just to do something seemingly simple, like parse a semantic version, make an api call and check the status code etc, etc. Maybe I keep making the wrong assumption that everyone is using the same tools the same way and thats why my opinions seem very strong. But I wouldn't even think of trying to…
Let’s say have a folder of tarballs and need to install the latest version. I could reach for an additional “dedicated” tool, get it installed into the CI environment and then incorporate it into the build process, or I could just make a slight modification to my existing shell script and do something like “ls mypkg-*.tar.gz | xargs -n1 | sort -Vr | head -n1” and then move on. But then we start publishing multiple release candidates and now I need to add to that logic to further distinguish between earlier and later rc versions. And so on and so forth…
Now I’m in agreement with you that this is a bad fit for shell scripting, but it is often pragmatic and expedient. And bc there is a cliff between bash and (say) python, some of the time, you’re going to choose the path of least resistance.
Now scale this out to a small team of engineers all facing the same dumb decision of needing to make some tradeoff when they would much rather be writing application logic. The lack of a ubiquitous and robust intermediate language leads to brittle CI fraught with security vulnerabilities.
While the example I provided is a bit contrived, this behavior isn’t hypothetical. I see it everywhere I’ve worked.
Re: I hate GitHub Actions with passion
#3041. Don't use bash, use a scripting language that is more CI friendly. I strongly prefer pwsh. 2. Don't have logic in your workflows. Workflows should be dumb and simple (KISS) and they should call your scripts. 3. Having standalone scripts will allow you to develop/modify and test locally without having to get caught in a loop of hell. 4. Design your entire CI pipeline for easier debugging, put that print state in, e…
We use Nuke for this purpose and I really like it. We're a .NET shop, and Nuke is C#, so works pretty well. Almost all our Nuke targets are shared between local and CI except some docker stuff that needs caching in CI.
Re: I hate GitHub Actions with passion
#305Earlier quoted context omitted.
I think in this case they hate the fact that they cannot easily SSH into the failing VM and debug from there. Like "I have to edit my workflow, push it, wait for it to run and fail, and repeat".
I had the weirdest thing happen a few days ago.. and only seemed to be happening in the GH runner for a PR.... somehow a function was duplicated in the action runner, and not in local or anwhere else... no idea how the corruption happened... it literally took me hours or pushing minor changes to try to correct the issue... I finally cat'd that file contents out and yep, the function was duplicated... no idea how. Had…
Obviously, by doing everything inside a container, where you control things. The GH action should only start the container, and pass it some env vars.
Re: I hate GitHub Actions with passion
#3061. Don't use bash, use a scripting language that is more CI friendly. I strongly prefer pwsh. 2. Don't have logic in your workflows. Workflows should be dumb and simple (KISS) and they should call your scripts. 3. Having standalone scripts will allow you to develop/modify and test locally without having to get caught in a loop of hell. 4. Design your entire CI pipeline for easier debugging, put that print state in, e…
1. Just no. Unless you are some sort of Windows shop.
I've never been a fan of PowerShell. I work across Windows and Linux on .NET and Nuke has become my go to tool for this.
Re: I hate GitHub Actions with passion
#307I think this post accurately isolates the single main issue with GitHub Actions, i.e. the lack of a tight feedback loop. Pushing and waiting for completion on what's often a very simple failure mode is frustrating. Others have pointed out that there are architectural steps you can take to minimize this pain, like keeping all CI operations isolated within scripts that can be run locally (and treating GitHub Actions fe…
I wonder what prevents a GH action from connecting to your VPN (Wireguard is fine) and post tons of diagnostics right onto your screen, and then, when something goes badly wrong, or when a certain point is reached, to just keep polling an HTTP endpoint for shell commands to execute.
I mean. I understand, it would time out eventually. But it may be enough time to interactively check a few things right inside the running task's process.
Of course this should only happen if the PR contains a file that says where to connect and when to stop for interactive input. You would only push such a file when an action is misbehaving, and you want to debug it.
I understand that it's a band-aid, but a band-aid is better than the nothing which is available right now.
Re: I hate GitHub Actions with passion
#308Earlier quoted context omitted.
Once you get beyond shell, make, docker (and similar), dependencies become relevant. At my current employer, we're mostly in TypeScript, which means you've got NPM dependencies, the NodeJS version, and operating system differences that you're fighting with. Now anyone running your build and tests (including your CI environment) needs to be able to set all those things up and keep them in working shape. For us, that i…
I use to have my Makefile call out and do `docker build ...` and `docker run ...` etc with a volume mount of the source code to manage and maintain tooling versions etc. It works okay, better than a lot of other workflows I have seen. But it is a bit slow, a bit cumbersome(for langs like Go or Node.js that want to write to HOME) and I had some issues on my ARM Macbook about no ARM images etc. I would recommend taking…
You mean nix inside a container? Or what exactly?
Re: I hate GitHub Actions with passion
#309What always surprises me in these discussions is how few people have used GitLab CI. What gives? It's far, far superior to GitHub and even existed first.
I found gitlab ci's yaml the smallest of the 3 I've used (gitlab, GitHub actions, CircleCI). But does gitlab ci have anything for sharing? GitHub actions are built around it, and CircleCI has orbs and contexts. For example, muse's guide for gitlab involves making your own container and managing the cache yourself (ref: https://mise.jdx.dev/continuous-integration.html#gitlab-ci ) GitHub actions is a couple of lines (r…
Not sure what you mean by "smallest". With GitLab you create docker images. That means you can easily run them locally and share them. In the example you gave the GitHub actions one looks like more lines of YAML but only works because they made an "action". If they had provided a Docker image then GitLab would be just as easy (and trivial to test locally).
Re: I hate GitHub Actions with passion
#310Can't you just have a good ole git repo in a server, use hooks to sync with github, and more hooks when you need 'actions' TM (C) (R)