Deploying Containers on NixOS: A Guide
bkiran.com
Deploying Containers on NixOS: A Guide
1–10 of 60 posts
Re: Deploying Containers on NixOS: A Guide
#2I suppose this is fine for a local machine setup. But, I would rather setup it a handful of VMs in a k8s cluster.
Currently, run my own k8s cluster with 20 worker nodes (basically just VMs on a few computers). Able to not only containerize my workloads but also evacuate workloads to different workers when I need to take down the server for maintenance (os updates, moving, kubectl upgrades).
I had actually planned to setup another remote cluster in my parents home (800 miles away), but ended up 86’ing that because their residential internet is the absolute worst.
Currently cluster only accessible when behind VPN or on local network. Haven’t setup proper authN/authZ controls yet.
Re: Deploying Containers on NixOS: A Guide
#3I suppose this is fine for a local machine setup. But, I would rather setup it a handful of VMs in a k8s cluster. Currently, run my own k8s cluster with 20 worker nodes (basically just VMs on a few computers). Able to not only containerize my workloads but also evacuate workloads to different workers when I need to take down the server for maintenance (os updates, moving, kubectl upgrades). I had actually planned to…
There is probably room for both. Planning to use a nix server as storage & DBs for my k8s cluster
Re: Deploying Containers on NixOS: A Guide
#4I suppose this is fine for a local machine setup. But, I would rather setup it a handful of VMs in a k8s cluster. Currently, run my own k8s cluster with 20 worker nodes (basically just VMs on a few computers). Able to not only containerize my workloads but also evacuate workloads to different workers when I need to take down the server for maintenance (os updates, moving, kubectl upgrades). I had actually planned to…
> Currently, run my own k8s cluster with 20 worker nodes (basically just VMs on a few computers).
Multiple worker VMs per physical host? Why?
Re: Deploying Containers on NixOS: A Guide
#5I suppose this is fine for a local machine setup. But, I would rather setup it a handful of VMs in a k8s cluster. Currently, run my own k8s cluster with 20 worker nodes (basically just VMs on a few computers). Able to not only containerize my workloads but also evacuate workloads to different workers when I need to take down the server for maintenance (os updates, moving, kubectl upgrades). I had actually planned to…
To each their own. When K8's is managed it's awesome.
I would like to do without the headache of dealing with K8's installation or some orchestrator layer. I can "schedule" my on application instances at the size I'm working with.
Also, I run this setup on cloud vm(multiple actually). So it's not restricted to running on a single machine running in a closet.
Re: Deploying Containers on NixOS: A Guide
#6I suppose this is fine for a local machine setup. But, I would rather setup it a handful of VMs in a k8s cluster. Currently, run my own k8s cluster with 20 worker nodes (basically just VMs on a few computers). Able to not only containerize my workloads but also evacuate workloads to different workers when I need to take down the server for maintenance (os updates, moving, kubectl upgrades). I had actually planned to…
There is probably room for both. Planning to use a nix server as storage & DBs for my k8s cluster
Nice, would love to know how it goes. What kind of storage and DB's are you planning to work with?
Re: Deploying Containers on NixOS: A Guide
#7NixOS OCI containers are a powerful way to run apps that are not packaged for Nix or NixOS. And because they’re ultimately systemd units, you can customize virtually everything without having to fiddle with the container runtime.
If you want to take this a step further and migrate or run a Compose project on NixOS, I maintain a tool that makes this pretty easy to do :)
Re: Deploying Containers on NixOS: A Guide
#8Could someone please help me understand the benefit of Nix versus just using docker-compose?
Also please don't tell me k8s it. Way too heavy.
I run a few applications on a souped-up Raspi (paperless-ngx, jellyfin, some postgres dbs). I can't see why this would improve things. :)
Honest question.
Re: Deploying Containers on NixOS: A Guide
#9Earlier quoted context omitted.
There is probably room for both. Planning to use a nix server as storage & DBs for my k8s cluster
Nice, would love to know how it goes. What kind of storage and DB's are you planning to work with?
Mostly trying to get rid of longhorn. I've found it to be a continuous source of troubles w/ etcd sync & IO issues. Current iteration of cluster is all optane so might give longhorn another go, but still want to move storage off cluster.
Especially source repo needs to live on some sort of striped zfs array. Ordered one of these quad nvme NAS things [0] so that's probably going to be storage. Either nix or proxmox...not decided.
Also still a bit fuzzy on what best game plan on PVs is. Minio/s3 or nfs appear to be options.
DB...just the usual suspects...mainly postgres for gitea I think. Mongo for dev stuff. Used to vanilla proxmox/docker/lxc so this is all unchartered territory for me.
Re: Deploying Containers on NixOS: A Guide
#10I suppose this is fine for a local machine setup. But, I would rather setup it a handful of VMs in a k8s cluster. Currently, run my own k8s cluster with 20 worker nodes (basically just VMs on a few computers). Able to not only containerize my workloads but also evacuate workloads to different workers when I need to take down the server for maintenance (os updates, moving, kubectl upgrades). I had actually planned to…
> Currently, run my own k8s cluster with 20 worker nodes (basically just VMs on a few computers). Multiple worker VMs per physical host? Why?
Mostly for easier segregation of workloads. Some of the IoT shit that runs on this cluster I segment it off through k8s and network policies. Also most workloads wouldn’t need to use all cores or memory on that machine (one machine has 128G)