Deploying Containers on NixOS: A Guide

Nix is a language, package manager, and OS. This post discusses NixOS.

While docker-compose allows you to compose your containers with a yaml/Dockerfiles, NixOS allows you to compose the system that all of your containers run on (from userspace down to kernel selection/configs, file system, etc), as well as your containers - all in a declarative .nix file. That .nix file can be used to spin up any number of identitically configured systems.

It's also reproducible, in that you can specify the sources (refined to a specific commit if you prefer) for any and all packages on the system - and build them with Nix within a sandboxed environment protecting dependencies and env configurations (Nix is also a powerful build system).

I've used docker-compose, k8s, and NixOS myself, being from a similar technical background as the author, but I find myself disagreeing with some of the author's opinions on the technologies. They're not wrong of course, but I've had different experiences.

k8s: Installing and using k8s can indeed be a nightmare. In my job, we use Azure, so it's not so bad since launching a cluster is mostly handled by Azure. Setting it up for personal use is less fun. The mountains of YAML you can end up using to deploy even semi-complex services is even less fun. That being said, I've been wanting to use it for a personal project (distributed cluster using cloud VPSs and bare metal at home connected using WireGuard). I just wish it was smaller and faster. Most guides recommend 2Gb of RAM and 2 CPU for the smallest of small deployments.

docker-compose: I actually love docker-compose for my personal stuff. I have an intel NUC hosting homeassistant, pihole, caddy, deluge, jellyfin and a handful of other stuff. Everything lives in a series of folders for each service. Backups (both data and code via git), disaster recovery, and just general reasoning about of it is so easy. The docker-compose files are small and easy to read. I also find docker-compose to be about as immutable as you'd like it - version control your docker-compose directories, pin your image SHAs, and you're in a good place. Or don't, and it will still work pretty well.

NixOS: I've done it. I installed it on my Framework Laptop since it was all the rage at the t ime. I lived with it for about a year, and it was okay - for day-to-day use - AFTER I had spent weeks learning how to use NixOS. I will freely admit it's an awesome technology in some respects. But the documentation just was not there. It was way too hard to learn how to do even basic tasks. I thought nix flakes might be the "aha" moment I was looking for, but I gave up trying to get that to work after a couple of days of troubleshooting. Don't even get me started on trying to package up something from scratch. As a random example, I googled "packaging python for nix" and the top result [1] is just way too complex for something that should be pretty simple. The example includes some abomination of a .nix file with inline bash and python scripts.

I don't really know where I'm going with this. I really do like the idea of NixOS. I just wish it was much, much easier to reason about. Curious to hear what others make of this.

[1](https://nixos.wiki/wiki/Packaging/Python)

> As a random example, I googled "packaging python for nix" and the top result [1] is just way too complex for something that should be pretty simple.

Aware that this is more of a critique about the documentation situation, as opposed to the python packaging situation. However, there is poetry2nix[1]. Which makes packaging look something like this:

    myPythonApp = mkPoetryApplication { projectDir = ./.; };

[1](https://github.com/nix-community/poetry2nix)(although it has been merged into nixpkgs master)

Nix has a dedicated following among some of the DevOps members in my company. Luckily, it stays mostly segregated, but I had an interesting conversation with some of them the other day about how awful troubleshooting Nix is. Now, mind you, these are pretty OG Nix contributors, and the response was basically: you have to want it enough.

I had my own falling out with Nix/NixOS over a year ago. I guess I didn't want it enough shrug.

For clarity, I still use home-manager because it's fairly painless, but anything above that is pretty much a no-go for me.

Yeah I agree it's manual but it takes about 5 minutes to SCP the password onto the machine.

I have some playbooks I setup to creating a new machine. All in all it takes about 10 min to get it up and running. Maybe not instant but at the moment I don't need anything else.