I am satisfied with pve and qm commands on proxmox, but I wish there was a nix answer to pfsense where my network and firewall are defined declaratively
Migrating from Proxmox to NixOS and Incus
51–60 of 73 posts
Re: Migrating from Proxmox to NixOS and Incus
#52Earlier quoted context omitted.
I agree!!! The "my entire infrastructure is defined in text files" alone tells OP never hear of Ansible.
So how are you reading this sentence? > While you can automate it with Terraform or Ansible, it..
>But fundamentally, Proxmox is built around clicking buttons. It is a GUI-first paradigm. While you can automate it with Terraform or Ansible, it often feels like fighting the tool. State drift is real. You change a setting in the UI to debug something, forget about it, and six months later your “infrastructure as code” is out of sync with reality.
Share the whole thing :)
Again, if you are managing everything as IaC, why touching the UI to make changes??
OP POV I just shared above tells everything that is wrong and Proxmox is not it.
Re: Migrating from Proxmox to NixOS and Incus
#53Earlier quoted context omitted.
https://github.com/Telmate/terraform-provider-proxmox ... I run Ansible against Proxmox hosts to ensure the config on them is what I want, and then Terraform all VMs into place on them. It's not too far off from having your own mini-EC2, minus some of the nice trappings like load balancers.
At $DAYJOB, we use the Telmate terraform proxmox provider, too. Upgrades over the years have had a few papercuts (mainly when values are shifted around), but it has overall been an immense timesaver. A terraform repo contains the 2 colocation datacenter proxmox environments plus a local lab edition of the same builds (locallab uses containers vs full VM's on the "real" hardware so that I can run almost all of the con…
Ah, I mean load balancing like an AWS ALB; obviously one can use an Nginx or HAproxy for that, but it's not a primitive, you need to deploy a machine and run it.
They have auto-balancing now? Damn. I wrote a balancer using the Google OR-SAT solver because there was no VMWare DRS equiv.
Re: Migrating from Proxmox to NixOS and Incus
#54Earlier quoted context omitted.
https://github.com/Telmate/terraform-provider-proxmox ... I run Ansible against Proxmox hosts to ensure the config on them is what I want, and then Terraform all VMs into place on them. It's not too far off from having your own mini-EC2, minus some of the nice trappings like load balancers.
Switched to that one some time ago, more features and adapts faster to changes in newer proxmox versions https://github.com/bpg/terraform-provider-proxmox
I will consider switching, but with like 400 VMs now it'd be a lot of work. Do you happen to know if it lets you deal with disks separately from the VM host? Used to use that flow on VMware, taint a machine but not its stateful secondary disk, and bring it back up with a fresh image and re-attach a database disk or whatever.
Re: Migrating from Proxmox to NixOS and Incus
#55Earlier quoted context omitted.
CLI is dandy but Terraform is even nicer. I run something like ten Proxmox clusters now around the world, and an AI agent can deploy a fully terraformed VM for me, tied into monitoring, asset mgmt, EDR, etc in a few minutes...
I run 20+… and it sure would be nice if Proxmox had a first party terraform provider. I have a lot of stuff I have to do via REST.
I concur highly. Well, make sure you buy your cheap support contract, help beta-test Proxmox Datacentre Manager, and push for it. They could probably just onboard one of the open source authors as a contractor or something if enough companies actually pony up the couple of grand per year per box. At 50K a server, paying like 2k for a proxmox license is a joke
Re: Migrating from Proxmox to NixOS and Incus
#56It's really hard to tell what the benefits of Incus over Ganeti are. The Incus site doesn't really lead with features or benefits that I can see. About the only things I could figure out are that Incus can manage containers and can manage instance creation through cloning a base volume, where ganeti (afaik) always creates volume and populates it. Well, and Incus seems to be MUCH more popular than Ganeti. I searched t…
Yeah, one thing I've found with Nix is that documentation is generally very terse and assumes you're capable of filling the blanks yourself, even though you might not know they exist.
Like... this... this is not great documentation (I know I know, contribute myself): https://wiki.nixos.org/wiki/Incus
Within a few lines:
> To provide non-root access to the Incus server, you will want to add your user to the incus-admin group. Don't forget to reboot.
I mean I get that they probably mean /etc/group, but going on from there plenty of examples of "just change the config to use x" or similar.
Re: Migrating from Proxmox to NixOS and Incus
#57I don't get their argument, I run GeForceNow on one of my Proxmox nodes in metal and there's no problem with it. It's a Debian underneath, after all. They didn't need to switch to NixOS to run Kodi.
Also, do they get PBS using ZFS snapshots? Do they get HA, live migration, shared storage, easy CephOS, easy snapshots, quick cloning? Do you really want to migrate a VM from one node to another using the command line when you're in some serious situation?
Sure, for a homelab this might be OK, but the UI does make things easy for a reason.and it's not a gimmick.
I agree on a lot of the points, though, I just set up a second cluster and it took over 3 work days because of how much repetitive work is needed to do so. To be able to just take a file with instructions, adjust it a bit and deploy would be so much easier.
Re: Migrating from Proxmox to NixOS and Incus
#58I am satisfied with pve and qm commands on proxmox, but I wish there was a nix answer to pfsense where my network and firewall are defined declaratively
This! pfSense/OPNSense seem to me like perfect candidates for declarative configuration. Even though, technically speaking, they are since everything is being save to a config file, which you can backup and restore. It's just less declarative then you'd like it to be. Sounds like they could easily employ Nix for handling that, though.
Re: Migrating from Proxmox to NixOS and Incus
#59I am satisfied with pve and qm commands on proxmox, but I wish there was a nix answer to pfsense where my network and firewall are defined declaratively
It is a scripting language but for Mikrotiks you can scp a .rsc file and
ssh $username@$routeraddr '/import file=setup.rsc'Re: Migrating from Proxmox to NixOS and Incus
#60Another proxmox-esque project I've been watching is Sylve , a control plane for FreeBSD that provides a web interface to jails, bhyve VMs, and containers [1]. It's new-ish, but it looks like a possible sweet spot replacement for both proxmox and TrueNAS (which was originally also built on FreeBSD before they switched to Linux IIRC), at least for my homelab-ing use case. Potentially eventually for environments at grea…
How does Sylve compare to Webmin [1]? [1]: https://docs.vultr.com/how-to-install-webmin-on-freebsd-14-0
Sylve, proxmox, and TrueNAS are all tools focused on: virtualization, containers, storage, and clusters. I've not used webmin, but it looks like a broader focus. Something like a kitchen sink of administering anything Unix-like via a web browser.
I wouldn't be surprised, glancing at webmin's site and docs, if it can do most/all of what more focused management tools like Sylve or proxmox can, but it seems to have so many other dials and knobs that IDK if I'd reach for webmin unless I were already familiar with it.
So Sylve is sort of the opposite of webmin in certain ways: highly focused, very new, built on a specific OS. It just looks interesting to me because FreeBSD has been great to work with already at my network gateway (running opnsense for firewall and routing), and I like the idea it replacing TrueNAS on a box I already own and use both for storage and running various self-hosted services (e.g. Jellyfin, Vaultwarden, Home Assistant, and the like) in containers and/or VMs as appropriate.