Show HN: Managed GitHub Actions Runners for AWS • Hacker News Board

Re: Show HN: Managed GitHub Actions Runners for AWS

#11

Post by toomuchtodo » Thu, Apr 04, 2024, 8:37 PM UTC

jacobwg wrote: post #9

toomuchtodo wrote: post #3

How will you compete if GitHub talks to the Azure folks (who have the benefit of Azure scale) and gets better compute and network treatment for runners? Or is the assumption GH running remains perpetually stunted as described (which is potentially a fair and legit assumption to make based on MS silos and enterprise inertia)? To be clear, this is a genuine question, as compute (even when efficiently orchestrated and a…

Yep, it's a good question! At the moment, my thoughts are roughly: GitHub's incentives and design constraints are different than ours. GitHub needs to offer something that covers a very large user-base, to cover the widest possible number of workflows, and they've done this by offering basic ephemeral VMs on-demand. CI and builds are also not GitHub's primary focus as an org. We're trying to be the absolute fastest p…

Good reply. It seems like you understand the market and where your product fits, which is half the battle.

Wishing you much success.

Re: Show HN: Managed GitHub Actions Runners for AWS

#12

Post by playingalong » Thu, Apr 04, 2024, 8:38 PM UTC

watermelon0 wrote: post #6

Hey, @jacobwg, this looks great. I couldn't find it anywhere on the page, but do you support Graviton3 (i.e. m7g instances) for GHA Runners? If the answer is no, are there any plans to support it in the future? > start them when an actual job request arrives, to keep job queue times around 5 seconds Did you have to fine-tune Ubuntu kernel/systemd boot to reach such fast startup times?

Not affiliated, just guessing.

This 5 seconds might be the warm start, not cold. I.e. they likely have a pool of autoscaled, multi tenant workers

Re: Show HN: Managed GitHub Actions Runners for AWS

#13

Post by timvdalen » Thu, Apr 04, 2024, 8:38 PM UTC

Congrats on shipping! We built something similar internally. Tweaking it for the right cost/availability/speed was interesting, but we now have it working to where workers are generally spun up from 0 faster than GitHub's own are.

Re: Show HN: Managed GitHub Actions Runners for AWS

#14

Post by jacobwg » Thu, Apr 04, 2024, 8:40 PM UTC

pestkranker wrote: post #5

How does it compare to BuildJet?

We're both offering managed GitHub Actions runners - some of the differences include:

- Depot runners are hosted in AWS us-east-1, which has implications for network speed, cache speed, access to internet services, etc. (BuildJet is hosted in Europe - maybe Hetzner?)

- Also thanks to AWS: each runner has a dedicated public IP address, so you're not sharing any third-party rate limits (e.g. Docker Hub) with other users

- We have an option to deploy the runners in your own AWS account or VPC-peer with your VPC

- We're integrating Actions runners with the acceleration tech we've built for container builds, starting with distributed caching

Re: Show HN: Managed GitHub Actions Runners for AWS

#15

Post by jacobwg » Thu, Apr 04, 2024, 8:48 PM UTC

watermelon0 wrote: post #6

Hey, @jacobwg, this looks great. I couldn't find it anywhere on the page, but do you support Graviton3 (i.e. m7g instances) for GHA Runners? If the answer is no, are there any plans to support it in the future? > start them when an actual job request arrives, to keep job queue times around 5 seconds Did you have to fine-tune Ubuntu kernel/systemd boot to reach such fast startup times?

We do support Graviton! I actually _just_ enabled them today, which we're calling "beta" for the moment: https://depot.dev/docs/github-actions/overview#depot-support....

The challenge with Arm is actually just that GitHub doesn't have a runner image defined for Arm. For the Intel runners, we build our image directly from GitHub's source[0], and we're doing the same for the Arm runners by patching those same Packer scripts for arm64. It also looks like some popular actions, like `actions/setup-*`, don't always have arm support either.

So the disclaimers for launching Depot `-arm` instances at the moment is basically just (1) we have no idea if our image is compatible with your workflows, and (2) those instances take a bit longer to start.

On achieving fast startup times, it's a challenge. :) The main slowdown that prevents a To address that at the moment, we do keep a pool of instances that boot once, load their volume contents, then shutdown until they're needed for a job. That works, at the cost of extra complexity and extra money - we're experimenting some with more exotic solutions now though like netbooting the AMI. That'll be a nice blog post someday I think.

[0] https://github.com/actions/runner-images/tree/main/images/ub...

Re: Show HN: Managed GitHub Actions Runners for AWS

#16

Post by crohr » Thu, Apr 04, 2024, 8:49 PM UTC

jacobwg wrote: post #14

pestkranker wrote: post #5

How does it compare to BuildJet?

We're both offering managed GitHub Actions runners - some of the differences include: - Depot runners are hosted in AWS us-east-1, which has implications for network speed, cache speed, access to internet services, etc. (BuildJet is hosted in Europe - maybe Hetzner?) - Also thanks to AWS: each runner has a dedicated public IP address, so you're not sharing any third-party rate limits (e.g. Docker Hub) with other user…

Yes, BuildJet runs from Hetzner - https://runs-on.com/reference/benchmarks-gha-providers/

Re: Show HN: Managed GitHub Actions Runners for AWS

#17

Post by jacobwg » Thu, Apr 04, 2024, 8:50 PM UTC

playingalong wrote: post #12

watermelon0 wrote: post #6

Hey, @jacobwg, this looks great. I couldn't find it anywhere on the page, but do you support Graviton3 (i.e. m7g instances) for GHA Runners? If the answer is no, are there any plans to support it in the future? > start them when an actual job request arrives, to keep job queue times around 5 seconds Did you have to fine-tune Ubuntu kernel/systemd boot to reach such fast startup times?

Not affiliated, just guessing. This 5 seconds might be the warm start, not cold. I.e. they likely have a pool of autoscaled, multi tenant workers

Yeah 5 seconds is from stopped to running, but to get that speed we need to pre-initialize the root EBS volumes so that they're not streaming their contents from S3 during boot. The GitHub Actions runner image is 50GB in size _just_ from preinstalled software!

Re: Show HN: Managed GitHub Actions Runners for AWS

#18

Post by jacobwg » Thu, Apr 04, 2024, 8:52 PM UTC

alas44 wrote: post #8

How do you ensure privacy/isolation between users if you have a pool of ready VMs that you re-use?

We don't re-use the VMs - a VM's lifecycle is basically:

1. Launch, prepare basic software, shut down

2. A GitHub job request arrives at Depot

3. The job is assigned to the stopped VM, which is then started

4. The job runs on the VM completes

5. The VM is terminated

So the pool exists to speed up the EC2 instance launch time, but the VMs themselves are both single-tenant and single-use.

Re: Show HN: Managed GitHub Actions Runners for AWS

#19

Post by crohr » Thu, Apr 04, 2024, 8:54 PM UTC

toomuchtodo wrote: post #3

How will you compete if GitHub talks to the Azure folks (who have the benefit of Azure scale) and gets better compute and network treatment for runners? Or is the assumption GH running remains perpetually stunted as described (which is potentially a fair and legit assumption to make based on MS silos and enterprise inertia)? To be clear, this is a genuine question, as compute (even when efficiently orchestrated and a…

I believe the solution is to decentralise, i.e. let the customer run the machines in their own AWS account (what I'm doing with RunsOn, link in bio if interested).

It is very hard for a single player to get favourable treatment from Azure / AWS / GCP to handle many thousands of jobs every day / hour.

I wish Depot all the luck, I think they've done good work wrt caching.

Re: Show HN: Managed GitHub Actions Runners for AWS

#20

Post by jacobwg » Thu, Apr 04, 2024, 9:02 PM UTC

werewrsdf wrote: post #7

I recently set up AWS Github runners with this terraform. It works well and you don't have to pay any extra in addition to AWS. https://github.com/philips-labs/terraform-aws-github-runner

Yeah this is a good option if you'd like something to deploy yourself! You can also build an AMI from GitHub's upstream image definition (https://github.com/actions/runner-images/tree/main/images/ub...) if you'd like it to match what's available in GitHub-hosted Actions.

With Depot, we're moving towards deeper performance optimizations and observability than vanilla GitHub runners - we've integrated the runners with a cache storage cluster for instance, and we're working on deeper integration with the compute platform that we built for distributed container image builds - as well as expanding the types of builds we can process beyond Actions and Docker, for instance.

But different options will be better for different folks, and the `philips-labs` project is good at what it does.