GitHub Actions: Ephemeral self-hosted runners and new webhooks for auto-scaling • Hacker News Board

Re: GitHub Actions: Ephemeral self-hosted runners and new webhooks for auto-scaling

#71

Post by twistedpair » Wed, Sep 22, 2021, 6:31 PM UTC

wcdolphin wrote: post #6

If anyone has experience using self hosted GH Actions at scale, I’d love to buy you a virtual coffee and hear about pros/cons for a parallelized CI flow currently running in Circle. Main motivation for switching would be simplification of tooling and increasing performance with better cache reuse and running within AWS for faster network access to ECR.

We moved all our builds (e.g. 100+) to GH Actions. We've been using GH Actions since it was a daily tar ball drop in a private GH Slack channel in Q3 2019.

Happy to answer any questions.

The biggest challenge has been the many GH Actions service outages/impacts. We're working on moving to self hosted runners to mitigate this.

Re: GitHub Actions: Ephemeral self-hosted runners and new webhooks for auto-scaling

#72

Post by twistedpair » Wed, Sep 22, 2021, 6:33 PM UTC

growse wrote: post #20

koalalorenzo wrote: post #19

I wish there was an official Helm Chart for k8s, like GitLab CI/CD Runner has, and not the kind that sits there and does no scale, but he kind that spins up workers on demand without taking too much resources while idle. I wish GitHub copied that feature from GitLab too!

It's not official, but there are K8s / github actions runner deployments: https://github.com/actions-runner-controller/actions-runner-... I've been playing about with this and it seems to work quite well. Startup latency is quite high, and it's one pod-per-job (I think), but seems pretty flexible.

I've been eyeing this for a while. My biggest hangup is that CI/CD is a major attack (e.g. supply chain) vector. If you use CI/CD for deploys, then a lot of highly privileged creds are in play.

I'd really prefer if GH made and managed the K8s operator (e.g. the most popular infra provisioning tool) themselves.

Re: GitHub Actions: Ephemeral self-hosted runners and new webhooks for auto-scaling

#73

Post by twistedpair » Wed, Sep 22, 2021, 6:34 PM UTC

vyrotek wrote: post #50

We're pretty happy with Azure DevOps on our team. But, these competing offerings between Azure and GitHub have been really confusing to follow. Especially since folks are pointing out that GitHub Actions is partly Azure DevOps under the hood. It just seems like a complicated branding play because some people will refuse to use an Azure service but will gladly use a GitHub service still owned by Microsoft?

What about the OSX runners? Those run in MacStadium, not Azure.

Re: GitHub Actions: Ephemeral self-hosted runners and new webhooks for auto-scaling

#74

Post by twistedpair » Wed, Sep 22, 2021, 6:36 PM UTC

xvilka wrote: post #53

The biggest problem with GitHub Actions that you can't restart just one job[1], it always restarts all jobs in the workflow. And this bug is not fixed for quite a while. Travis CI and Appveyor both allow that, of course. [1] https://github.com/actions/runner/issues/432

And, if you restart a job, no new entry is made for the job history, so it just overwrites the job history on rerun.

This is likely because they modeled the jobs as one-per-job-def-and-commit, so they don't have a UX to show two.

This is a security blind spot, since you can do something naughty in a job, then rerun it and the logs are no longer accessible.