Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LXD containers on macOS at near-native speeds

beringresearch.github.io

Post reply on HN
Print view View original

91–100 of 134 posts

Re: LXD containers on macOS at near-native speeds

#91

Post by cmeacham98 »

LoganDark wrote: post #90
VWWHFSfQ wrote: post #84

Earlier quoted context omitted.

I ended up just getting a System 76 Meerkat computer for doing anything locally with LXC/Docker. Linux emulation in Windows or macOS is unbearably bad if you really need to do serious work.

Even systemd will run in Docker on macOS. Can you clarify what makes macOS unacceptable for emulating Linux?

Using a VM for Docker gives you the worst of both worlds.

You get all the downsides of a VM (persistent wasted RAM/disk space, awful disk IO performance, etc) AND all the downsides of containers (difficult to introspect when your minimal image has no shell, janky networking, etc).

On recent M1/M2 Macs you take the additional pain (and thus performance hit) of translation to x86 (because that's almost certainly what your containers are in.

Re: LXD containers on macOS at near-native speeds

#92

Post by hamandcheese »

MuffinFlavored wrote: post #61
hamandcheese wrote: post #50

Earlier quoted context omitted.

Because docker for Mac has always kind of sucked, I’ve moved on to making nix-based development environments and I’ve been very pleased so far. Lately I’ve been trying https://devenv.sh/ and it works great! I haven’t tried it for ruby, though I have used vanilla nix shell for ruby projects before and it worked quite well after I over-rode GEM_PATH and GEM_HOME to the correct values.

help me understand nix file helps you build OCI image and then you run it not locally/somewhere else? can that be compared apples to apples to docker where you can build with Dockerfile syntax, and then run locally the image?

Nix builds native software for your platform reproducibly. There is no OCI image, no Docker, just a way to reliably install software, which is a great foundation for building per-project development environments.

(Aside: you can build oci images using nix, but that’s probably what you would use for building deployable artifacts for production, not for development).

I would not say it is comparable with a Dockerfile - a Dockerfile is closer to a shell script, with caching between each step. Credit to Dockerfile syntax for being very easy to grok, but it is inherently not reproducible.

The perceived reliability of docker for development probably comes from the fact that you can share a prebuilt image, but rebuilding might not always work.

Re: LXD containers on macOS at near-native speeds

#93

Post by hamandcheese »

bsnnkv wrote: post #73
dchuk wrote: post #40

So is there some canonical guide to running a docker compose style app on Mac m1 machines that has good filesystem performance? It seems like there’s many ways to approach the topic now so it’s hard to tell which one is “winning”. I’d love to containerize all of my local development efforts (scripts and rails apps) but the slow ass filesystem always ruined it in the past.

I've recently switched to using nix-shell and overmind with a Procfile to officially retired docker-compose from all of my projects. Such a breath of fresh air. Like Mitchell Hashimoto (Hashicorp), I no longer do any development on macOS directly; I have a Linux VM where all my projects live and have their environments managed with a shell.nix file. Highly highly recommend it.

If you’re using nix, why are you using a VM? Nix on macOS works pretty well these days.

Re: LXD containers on macOS at near-native speeds

#94

Post by bsnnkv »

unboxingelf wrote: post #76
bsnnkv wrote: post #73

Earlier quoted context omitted.

I've recently switched to using nix-shell and overmind with a Procfile to officially retired docker-compose from all of my projects. Such a breath of fresh air. Like Mitchell Hashimoto (Hashicorp), I no longer do any development on macOS directly; I have a Linux VM where all my projects live and have their environments managed with a shell.nix file. Highly highly recommend it.

Would you elaborate on the setup? I recall seeing a comment or article where Mitchell mentioned this setup previously. I’ve been using Make + Docker for development on macOS for a long time now and perhaps it’s time for a change. I’ve looked at nix once or twice over the years but it never really clicked.

Sure, this is a super abbreviated version of my current setup for developing https://notado.app

shell.nix - this brings the packages I need to develop into the $PATH when I run `nix-shell` in the repo

    {pkgs ? import  {}}:
    with pkgs;
      mkShell {
        buildInputs = [
          meilisearch
          openssl
          overmind
          pkg-config
          postgresql
          rustup
          tmux
        ];

        shellHook = ''
          export OVERMIND_PROCFILE=Procfile.dev
          export PGDATA="data.pg"
          export MEILI_DB_PATH="data.ms"
        '';
      }
Procfile.dev - this is used to start the background services I need, the format is "name: command"

    postgres: postgres -K /tmp
    meilisearch: meilisearch
Then I can run `nix-shell` to load all the dependencies and `overmind start` to bring up Postgres + Meilisearch with ignored data dirs relative to the project folder. If any of these packages need to be pinned to a specific version you can do that with this tool[1].

That's pretty much it - with all the background services running you can load your monolith or microservices with cargo/npm/etc. It's all very nice and clean!

[1]: https://lazamar.co.uk/nix-versions/ - I actually just used this in a new shell.nix for my personal website to pin a version of Hugo from 2019 (when I last touched it) so that it would build again

Re: LXD containers on macOS at near-native speeds

#95

Post by saagarjha »

codedokode wrote: post #59
Veliladon wrote: post #9

Earlier quoted context omitted.

QEMU can use the MacOS virtualization framework layer to run a VM built for the same arch without going through any translation. From there you can register Rosetta as a runtime and use it to run x86-64 Linux apps on ARM64 if you want. ( https://developer.apple.com/documentation/virtualization/run... )

I don't understand why one needs to emulate a virtual machine to run a container. Why do you need a kernel? Why emulate physical devices, interrupts and everything else? Why cannot one run a container as a protected mode application?

It’s difficult to emulate a kernel. You can try and there have been multiple attempts to do so with varying levels of success but most have pitfalls that people eventually run into.

Re: LXD containers on macOS at near-native speeds

#96

Post by saagarjha »

jbverschoor wrote: post #25
ricardobeat wrote: post #22

Earlier quoted context omitted.

Not if you’re running x86 emulation on an ARM host. It’s terribly slow. EDIT: This is from experience. When I started using a M1 mac, our docker builds on x86 images took up to 5x or 10x longer vs running on an arm64 image. Had slightly faster, but similar results running my own docker host on Lima, or podman, or building without docker inside a UTM host which also uses qemu.

I don’t think that’s true. https://developer.apple.com/documentation/virtualization/run...

Requires an arm64 kernel. Rosetta in this context essentially replaces usermode QEMU.

Re: LXD containers on macOS at near-native speeds

#97

Post by saagarjha »

cpuguy83 wrote: post #78
a-dub wrote: post #77

Earlier quoted context omitted.

you have to add support for your infrastructure to support building of everything for two architectures. sometimes problems are architecture dependent. if you have to use x86 containers for some reason, they will be dog slow because you'll be running the kernel under emulation and in userland rosetta won't help.

Rosetta absolutely does help. You register rosetta (which can be mounted into the VM using macOS's virtualization framework) with binfmt_misc and your x86 binaries inside the VM will run through Rosetta.

Rosetta doesn’t help here because the kernel would be ARM, and the parent commenter wants an x86-64 kernel.

Re: LXD containers on macOS at near-native speeds

#98

Post by saagarjha »

pxc wrote: post #38

At first, I thought this was based on a syscall compatibility layer like Solaris' Linux Zones or WSL1 (RIP), or the Linux support in FreeBSD and NetBSD. If you've ever tried to spin up a whole bunch of Docker containers in WSL2 and watched `vmmem` memory and CPU usage explode, you know that 'near-native speed' in VMs comes with lots of asterisks. Does macOS have usable native macOS containers yet?

No.

Re: LXD containers on macOS at near-native speeds

#99

Post by saagarjha »

sirwhinesalot wrote: post #48
eurasiantiger wrote: post #28

Earlier quoted context omitted.

Can confirm this works, but some applications will be slow. Seen a magnitude of difference with x86 JVM containers between Rosetta 2 and native ARM64.

Rosetta 2 works great for AoT stuff but for JIT you'll always end up with at least a 2x slowdown since it needs to always JIT the JVMs JIT output. That said, it working at all is pretty amazing.

This doesn’t mean there’s a 2x slowdown. (I’ve seen 1.5x to 5x.)

Re: LXD containers on macOS at near-native speeds

#100

Post by cpuguy83 »

saagarjha wrote: post #97
cpuguy83 wrote: post #78

Earlier quoted context omitted.

Rosetta absolutely does help. You register rosetta (which can be mounted into the VM using macOS's virtualization framework) with binfmt_misc and your x86 binaries inside the VM will run through Rosetta.

Rosetta doesn’t help here because the kernel would be ARM, and the parent commenter wants an x86-64 kernel.

X86 userspace on an arm kernel is exactly what Rosetta would be used for.
Post reply on HN