Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Cross-Platform GitHub Action

github.com

Post reply on HN
Print view View original

21–29 of 29 posts

Re: Show HN: Cross-Platform GitHub Action

#21

Post by kylegalbraith »

adobrawy wrote: post #5

I previously tried to use Docker `docker/setup-qemu-action@v2` and `docker/setup-buildx-action@v2` for this purpose (see that example https://github.com/docker/build-push-action#git-context ). Thanks to buildkit, platform switching works transparently. However, building on ARM via QEMU on GitHub Actions is terribly slow (something like 5 times more), which is hard to accept. Therefore, full of hope, I am waiting for…

This was one of the frustrations that led us to create Depot (https://depot.dev/). It's a remote Docker build service that launches Intel & Arm VMs to build native multi-platform images in parallel on native CPUs with zero emulation.

Behind the scenes we are running the same engine as Docker, BuildKit, on VMs in AWS for both architectures. We combine native architecture support with a 50GB persistent SSD cache that is automatically available across builds, so no more saving/loading cache over the network.

All those things combined makes multi-platform or Arm image builds in GitHub Actions ~15x faster using Depot (example: https://depot.dev/benchmark/temporal). And it's a one line change, swap `docker build` for `depot build` or use our `depot/build-push-action` instead of `docker/build-push-action`.

Re: Show HN: Cross-Platform GitHub Action

#22

Post by JacobCarlborg »

adobrawy wrote: post #5

I previously tried to use Docker `docker/setup-qemu-action@v2` and `docker/setup-buildx-action@v2` for this purpose (see that example https://github.com/docker/build-push-action#git-context ). Thanks to buildkit, platform switching works transparently. However, building on ARM via QEMU on GitHub Actions is terribly slow (something like 5 times more), which is hard to accept. Therefore, full of hope, I am waiting for…

Yeah, I've been using `docker/setup-qemu-action` as well to run on non-x86-64 architectures for Linux [1]. But since it's Docker it's Linux only and doesn't support other operating systems.

GitHub has Apple Silicon on the roadmap, but IIRC it was at the end of this year. They already support Apple Silicon for self hosted runners [2].

[1] https://github.com/jacob-carlborg/lime/blob/f4d9c8c4265b61b2...

[2] https://github.blog/changelog/2022-08-09-github-actions-self...

Re: Show HN: Cross-Platform GitHub Action

#24

Post by adobrawy »

kylegalbraith wrote: post #21
adobrawy wrote: post #5

I previously tried to use Docker `docker/setup-qemu-action@v2` and `docker/setup-buildx-action@v2` for this purpose (see that example https://github.com/docker/build-push-action#git-context ). Thanks to buildkit, platform switching works transparently. However, building on ARM via QEMU on GitHub Actions is terribly slow (something like 5 times more), which is hard to accept. Therefore, full of hope, I am waiting for…

This was one of the frustrations that led us to create Depot ( https://depot.dev/ ). It's a remote Docker build service that launches Intel & Arm VMs to build native multi-platform images in parallel on native CPUs with zero emulation. Behind the scenes we are running the same engine as Docker, BuildKit, on VMs in AWS for both architectures. We combine native architecture support with a 50GB persistent SSD cache that…

Are you SOC2 compliant? Can an image built in this way be loaded back (`load: true`) to CI without OCI registry?

Re: Show HN: Cross-Platform GitHub Action

#25

Post by dbingham »

Arnavion wrote: post #15
dbingham wrote: post #13

Earlier quoted context omitted.

The architecture of the docker image matters. If you build a docker image on an ARM machine (like a m1 or m2 Mac) you can't run it in x86 architecture (like a T3 or M5 AWS instance). If you build on x86 architecture, you can't run it on ARM architecture. That is, unless you use Docker's cross platform build capability (buildx). Which was still considered experimental the last time I looked at it (about a year ago).

You can build other-arch images with regular `build`. You'll of course need QEMU hooked up through binfmt to be able to execute `RUN` steps while building the image, but you can do that yourself without involving `buildx`.

Neat! I need to read up on QEMU. I haven't looked into cross arch images in a while.

Re: Show HN: Cross-Platform GitHub Action

#26

Post by kylegalbraith »

adobrawy wrote: post #24
kylegalbraith wrote: post #21

Earlier quoted context omitted.

This was one of the frustrations that led us to create Depot ( https://depot.dev/ ). It's a remote Docker build service that launches Intel & Arm VMs to build native multi-platform images in parallel on native CPUs with zero emulation. Behind the scenes we are running the same engine as Docker, BuildKit, on VMs in AWS for both architectures. We combine native architecture support with a 50GB persistent SSD cache that…

Are you SOC2 compliant? Can an image built in this way be loaded back (`load: true`) to CI without OCI registry?

We aren't SOC2 compliant yet, but we are planning to do that this year. Yup, you can pass `depot build --load` (the same parameters you use with buildx) and we will return the built image to you. Or you can use `--push` and we will push it up to your registry.

Here is my direct email if anyone wants to chat/learn/share more about this problem space, kyle [at] depot.dev.

Re: Show HN: Cross-Platform GitHub Action

#27

Post by Arnavion »

dbingham wrote: post #25
Arnavion wrote: post #15

Earlier quoted context omitted.

You can build other-arch images with regular `build`. You'll of course need QEMU hooked up through binfmt to be able to execute `RUN` steps while building the image, but you can do that yourself without involving `buildx`.

Neat! I need to read up on QEMU. I haven't looked into cross arch images in a while.

For systemd distros, you might already have systemd-binfmt.service in the systemd package, which when started will automatically register all architectures specified in config files under /usr/lib/binfmt.d with the kernel. Then your qemu package will probably contain one config file in that directory for every arch. So if you enable and start the systemd-binfmt service, you'll automatically have a bajillion architectures registered with the kernel to run under qemu and you don't have to do anything else.

Or you can register manually by writing to /proc/sys/fs/binfmt_misc/register . Note that you'll want to register the statically linked version of qemu (qemu-user-static or whatever your distro calls it), and that you'll want to use at least the O and F flags so that the binary works inside containers automatically insead of needing to be mounted from the host.

Re: Show HN: Cross-Platform GitHub Action

#28

Post by mananaysiempre »

Arnavion wrote: post #27
dbingham wrote: post #25

Earlier quoted context omitted.

Neat! I need to read up on QEMU. I haven't looked into cross arch images in a while.

For systemd distros, you might already have systemd-binfmt.service in the systemd package, which when started will automatically register all architectures specified in config files under /usr/lib/binfmt.d with the kernel. Then your qemu package will probably contain one config file in that directory for every arch. So if you enable and start the systemd-binfmt service, you'll automatically have a bajillion architect…

> For systemd distros, you might already have systemd-binfmt.service in the systemd package, which when started will automatically register all architectures specified in config files under /usr/lib/binfmt.d with the kernel. Then your qemu package will probably contain one config file in that directory for every arch. So if you enable and start the systemd-binfmt service, you'll automatically have a bajillion architectures registered with the kernel to run under qemu and you don't have to do anything else.

Note that this can screw up other kinds of builds, as the autoconf check for cross-compilation relies on a cross-compiled executable being unrunnable. (Seen with a wine binfmt handler and a MinGW cross compiler.)

Re: Show HN: Cross-Platform GitHub Action

#29

Post by ben-schaaf »

Arnavion wrote: post #20
ben-schaaf wrote: post #19

Earlier quoted context omitted.

Mixing compilers is not a problem because they use the same standard ABI. You often need to link against the oldest supported version of a library, or dlsym what you need, but you certainly don't need to compile per-distro. The only big exception I know of is musl. We've been happily compiling a single executable using ~latest clang for years with very few problems beyond using a symbol that doesn't exist on older di…

>Mixing compilers is not a problem because they use the same standard ABI. So first, you said: >>What libraries on Linux don't use the SystemV C ABI on x86? ... even though the conversation was about ARM. Second, you seem to assume that there could be no other problems other than ABI, even though... >with very few problems beyond using a symbol that doesn't exist on older distros. ... Oh, I shouldn't even need to exp…

> even though the conversation was about ARM.

Apologies, I wanted to be specific which standard is being followed. Nonetheless afaik the ABI is the same for all arm distros.

>> with very few problems beyond using a symbol that doesn't exist on older distros. > ... Oh, I shouldn't even need to explain it, because you're already aware of it.

A missing symbol is a problem regardless of where you're compiling. If you're linking against something that doesn't exist it'll fail; whether that's on your client's machine running an old distro or when you're compiling on that old distro yourself. You could argue it's better to fail early here, but the work in fixing this bug is approximately the same.

> I'm amazed I even have to argue about such basic knowledge. If you don't believe me, maybe you'll believe OSDev?

OSDev is incorrect here. Take it from GNU themselves: https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html. Symbols in libgcc are versioned; either your code runs and links against the exact correct version or it fails. If you want to run on old distros with libgccs that don't have newer symbols either use an older gcc, statically link libgcc or use clang.

Post reply on HN