LXC and LXD: a different container story

Even my mail server runs in LXC, and it is a direct descendant of a Red Hat 5 server from ~20 years ago!

I've often seen LXD vs Docker described as LXD being better suited to running distros and distro-like environments, with Docker being better suited to single-application environments.

So that's what I do. For many of dev and testing environments, there's an LXD container, which I use over SSH/rsync daily. Compiles, long-running tests, bandwidth heavy network actions and so on are done in those containers, on a few fast servers in data centres. It's much faster than my laptop.

One motivation for running Linux in LXD containers instead of on bare metal was to allows me to decouple changes to host OS version and networking from the containers in which I do most my work. Previously I used bare metal, i.e. ran Linux on servers, but found it annoying that I couldn't update the host OS and especially the kernel or disto major version, without shutting down everything in Screen sessions and long-lived networking tests that are not so easy to shut down and restart (without a rewrite anyway).

Being containers, they can use host filesystems and devices almost directly. So they are great for things like I/O performance tests, with confidence that it's basically testing host performance and behaviour.

Unfortunately, after getting deep into both those things, I found LXD wasn't quite as run-a-distro friendly as it first appeared, and it wasn't as reliable at replicating host I/O performance either.

Doing file I/O with "shift=true" host-filesystem mounts in LXD (the most sensible mode) turns out to be have very much slower O_DIRECT performance than it should, so that screwed up my database storage tests unexpctedly until I realised. Now I run low-level storage performance tests outside LXD, because I don't trust it.

As for distro-like environments, I eventually found stateful snapshot+restore or migation of dev and test environments is permanently broken. No LXD container I've used over many years has ever successfully been able to live-snapshot/migrate, without hitting an error which prevents it. This isn't some obscure bug, either. It's never worked, and my browsing of forums and issue trackers leads to the view that it's not actually expected to work for almost any distro-like environment, despite being one of the headline features.

As a result, one of the main factors motivating using LXD instead of bare metal for me turned out not to work. It's always possible to shut down a container, but that loses so much state, long-lived Screen sessions, running processes and so on that it's about as disruptive as updating bare metal. I.e. no real advantage.

I could switch back to VMs, which are excellent for snapshotting and migration, as I used to use (with libvirt+kvm) but they have a different problem for much of my work: Host filesystem/blockdev sharing is relatively slow. Not only could I not use them to measure performance against various kernels on real hardware (it would be measuring the VM as much as anything), I also run many data-intensive jobs, and the closer I can get to those host storage devices, the better.

The last reason it's felt buggy and rough is when removing a host-container filesystem mount from a container, it has often deleted the mount point on the host as well, disrupting other processes using it. Operations on the container are not supposed to change the host itself. You get used to working around this, but it's annoying.

Ah well, nothing's perfect. It's still a very useful tool, with some rough edges. I'd look into fixing the I/O performance and host-unmounting bug if I thought the snapshot/migration feature would be made to work someday, but because that looks unlikely, plus issues that came with Snap, I'm not as motivated and will live with hackish workarounds.

For things like my mail, web, SpamAssassin and other services, LXD has been great, and the issues I encountered aren't really a problem. This is the sort of thing which Docker is pretty good for as well. However, due to history I've tended to keep servers going for a long time (with my mail server winning the crown as it's an image that's been gradually modified and upgraded for about 20 years, across many different ways of running Linux in container-like environments, starting with chroot). LXC/LXD is better than Docker for this use case, though either can be made to work.