Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Managed GitHub Actions Runners for AWS

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 59 posts

Re: Show HN: Managed GitHub Actions Runners for AWS

#41

Post by liamfd »

jitl wrote: post #33

At Notion we run our GitHub Actions jobs on ECS, and use auto-scaling to add and remove hosts from the ECS cluster as demand fluctuates throughout the day. We also age out and terminate hosts although they usually live for a few days to a week. I guess we had to pay some one time setup costs around configuring the ECS cluster and fiddling runner tags, but it seems to work pretty well. We have our own cache action alt…

This sounds like a good setup - if you don't mind me asking, what do you use for your auto scaling metric?

Also curious how much y'all isolate it from your other infra. I've thought about this but I've been torn on whether I'd set up a separate vpc for it.

Re: Show HN: Managed GitHub Actions Runners for AWS

#42

Post by SOLAR_FIELDS »

One of the most interesting value adds for me is not any of the things mentioned by OP. I would like to have a managed hosted runner solution where I can have a buildkit cache also in the same data center that I manage where I don’t have to pay ingress/egress to that cache but also I don’t have to manage my own runner infra. I have done the whole self hosted Karpenter + Actions Runner Controller thing to achieve this and it is a lot of work to set up and tune to get right.

The problem is actually really that GitHub’s caching offering is very limited for anything except the most basic of use cases and also they don’t offer a way to colo your own cache with them so that you aren’t paying cloud fees back and forth. You have to use their machines, their storage and their protocol which is only really viable if your definition of caching is literally just “upload files here” and “check if the uploaded built file already exists”.

Yes, I’m aware that buildkit offers “experimental” GHA caching support. But given how fat image layers are it’s basically unusable for anything beyond a toy project that builds a couple layers on top of an alpine image (as of the time of writing this post GHA limits cache size to 10gb per repo. Fine if you’re building npm or pypi packages or whatever, but hilariously inadequate for buildkit layer caching)

Re: Show HN: Managed GitHub Actions Runners for AWS

#43

Post by boundlessdreamz »

SOLAR_FIELDS wrote: post #42

One of the most interesting value adds for me is not any of the things mentioned by OP. I would like to have a managed hosted runner solution where I can have a buildkit cache also in the same data center that I manage where I don’t have to pay ingress/egress to that cache but also I don’t have to manage my own runner infra. I have done the whole self hosted Karpenter + Actions Runner Controller thing to achieve this…

The depot.dev service has excellent caching for docker. It's almost like building locally.

Though the site (depot.dev) focuses on that aspect, this post doesn't.

@jacobwg - Do the runners in AWS get the same docker caching performance as depot.dev hosted runners?

Re: Show HN: Managed GitHub Actions Runners for AWS

#44

Post by crohr »

SOLAR_FIELDS wrote: post #42

One of the most interesting value adds for me is not any of the things mentioned by OP. I would like to have a managed hosted runner solution where I can have a buildkit cache also in the same data center that I manage where I don’t have to pay ingress/egress to that cache but also I don’t have to manage my own runner infra. I have done the whole self hosted Karpenter + Actions Runner Controller thing to achieve this…

What you are looking for is a local S3 cache, which buildx supports as a backend. Just make sure you have an S3 gateway connected to your VPC (and that your S3 bucket is in the same region than your runners!) and enjoy free bandwidth, unlimited cache size, and crazy fast network throughput.

https://runs-on.com/reference/caching/ https://runs-on.com/features/s3-cache-for-github-actions/

Re: Show HN: Managed GitHub Actions Runners for AWS

#45

Post by SOLAR_FIELDS »

crohr wrote: post #44
SOLAR_FIELDS wrote: post #42

One of the most interesting value adds for me is not any of the things mentioned by OP. I would like to have a managed hosted runner solution where I can have a buildkit cache also in the same data center that I manage where I don’t have to pay ingress/egress to that cache but also I don’t have to manage my own runner infra. I have done the whole self hosted Karpenter + Actions Runner Controller thing to achieve this…

What you are looking for is a local S3 cache, which buildx supports as a backend. Just make sure you have an S3 gateway connected to your VPC (and that your S3 bucket is in the same region than your runners!) and enjoy free bandwidth, unlimited cache size, and crazy fast network throughput. https://runs-on.com/reference/caching/ https://runs-on.com/features/s3-cache-for-github-actions/

Looks neat but is there a way to guarantee that it’s colo’ed with GHA hosted runners and that I won’t pay ingress/egress? If not then I don’t see how it’s much different than simply putting up my own bucket aside from saving me the logistics around permissions etc.

Edit: I see. This solution you linked doesn’t use GHA hosted runners at all - it’s intended to be a turnkey self hosted runner solution. In other words, a direct competitor to the service linked in OP. That wasn’t super clear from your comment but after reading your links it is more clear. I do really like the pricing here, if it actually works as advertised it’s a pretty great value prop for a lot of orgs.

Re: Show HN: Managed GitHub Actions Runners for AWS

#46

Post by crohr »

SOLAR_FIELDS wrote: post #45
crohr wrote: post #44

Earlier quoted context omitted.

What you are looking for is a local S3 cache, which buildx supports as a backend. Just make sure you have an S3 gateway connected to your VPC (and that your S3 bucket is in the same region than your runners!) and enjoy free bandwidth, unlimited cache size, and crazy fast network throughput. https://runs-on.com/reference/caching/ https://runs-on.com/features/s3-cache-for-github-actions/

Looks neat but is there a way to guarantee that it’s colo’ed with GHA hosted runners and that I won’t pay ingress/egress? If not then I don’t see how it’s much different than simply putting up my own bucket aside from saving me the logistics around permissions etc. Edit: I see. This solution you linked doesn’t use GHA hosted runners at all - it’s intended to be a turnkey self hosted runner solution. In other words, a…

Oh yes, it can't work with GHA hosted runners otherwise you'll pay egress fees. From your first post I was assuming you were starting from the point of view that you would be running your own runners already.

It does work as advertised, try it :) And yes RunsOn is a direct competitor to the 5 YCombinator-funded companies operating in this space (Ubicloud, Warpbuild, Buildjet, Blacksmith, Depot).

Re: Show HN: Managed GitHub Actions Runners for AWS

#47

Post by kylegalbraith »

SOLAR_FIELDS wrote: post #42

One of the most interesting value adds for me is not any of the things mentioned by OP. I would like to have a managed hosted runner solution where I can have a buildkit cache also in the same data center that I manage where I don’t have to pay ingress/egress to that cache but also I don’t have to manage my own runner infra. I have done the whole self hosted Karpenter + Actions Runner Controller thing to achieve this…

Other Depot founder here. This is a really great point.

You've hit on all the main points regarding Docker image cache in GHA. Persisting the massive layer cache over networks is incredibly slow and has weird limits (like 10GB per repo). We persist the layer cache to ceph volumes and orchestrate your cache so it's immediately available across builds with our first service, accelerated container image builds. Our GHA runners run right next to that same infra, so you don't have ingress/egress. All that can be hosted in your own AWS account (we're also open to running that in any general compute environment for folks who need it).

Re: Show HN: Managed GitHub Actions Runners for AWS

#48

Post by jacobwg »

SOLAR_FIELDS wrote: post #42

One of the most interesting value adds for me is not any of the things mentioned by OP. I would like to have a managed hosted runner solution where I can have a buildkit cache also in the same data center that I manage where I don’t have to pay ingress/egress to that cache but also I don’t have to manage my own runner infra. I have done the whole self hosted Karpenter + Actions Runner Controller thing to achieve this…

Hey, yep we have this today! Depot's original product is a fully-managed container build service (https://news.ycombinator.com/item?id=34898253) that caches all BuildKit layers to SSDs, so there's no cache-to/cache-from and cache doesn't need to transfer over the network at all.

Our original version of that system used vanilla BuildKit + EBS volumes + orchestration, nowadays we've replaced EBS with a distributed ceph storage cluster for significantly faster IOPS and throughput and have modified BuildKit to be better suited for high-performance distributed builds.

Both the container build service and the Actions runners are in the same AWS VPC, so they get good network performance between the two and don't need to egress over the internet.

Re: Show HN: Managed GitHub Actions Runners for AWS

#49

Post by jacobwg »

boundlessdreamz wrote: post #43
SOLAR_FIELDS wrote: post #42

One of the most interesting value adds for me is not any of the things mentioned by OP. I would like to have a managed hosted runner solution where I can have a buildkit cache also in the same data center that I manage where I don’t have to pay ingress/egress to that cache but also I don’t have to manage my own runner infra. I have done the whole self hosted Karpenter + Actions Runner Controller thing to achieve this…

The depot.dev service has excellent caching for docker. It's almost like building locally. Though the site (depot.dev) focuses on that aspect, this post doesn't. @jacobwg - Do the runners in AWS get the same docker caching performance as depot.dev hosted runners?

At the moment, I think you'd want to use both products together, i.e. using `depot build` in place of `docker build` to move the container build portion to a Depot container builder.

I'd like to have a more automatic integration at some point - the challenge is that a lot of BuildKit's architecture performs best when many different build requests all arrive at a single build host, it is then able to efficiently deduplicate and cache work across all those build requests. So you really want the many different Actions jobs all communicating with the same BuildKit host.

We have some ideas for reducing the amount of change to Actions workflows to adopt ^ - longer term we're also working on our own build engine, to free those workloads from being confined to single hosts (be that single CI runners or single container builders).

Re: Show HN: Managed GitHub Actions Runners for AWS

#50

Post by jacobwg »

LilBytes wrote: post #28

Hey Jacob, awesome suggestion! Are you building your base image from the GitHub runner-images repo? Do you have any appetite for building self hosted EC2 agents for Azure DevOps and GitHub? I'm happy to help if you are, I'm working on something similar myself for my employer.

Hey, we are building our base image from the runner-images repo! I'll send you an email!
Post reply on HN