Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Migrating from Proxmox to NixOS and Incus

nijho.lt

Post reply on HN
Print view View original

41–50 of 73 posts

Re: Migrating from Proxmox to NixOS and Incus

#41

Post by wingmanjd »

mikestorrent wrote: post #20
whalesalad wrote: post #12

I haven't abandoned Proxmox yet, but the take here resonates with me. I do not like configuring appliances. I prefer defining infra as code, having that diffable, assertable, etc. I have had pretty good luck managing Proxmox clusters with the Proxmox API ( https://pve.proxmox.com/pve-docs/api-viewer/index.html ) or just letting the agent shell in as root (lol). I built a very simple provisioning tool called vmfactory…

https://github.com/Telmate/terraform-provider-proxmox ... I run Ansible against Proxmox hosts to ensure the config on them is what I want, and then Terraform all VMs into place on them. It's not too far off from having your own mini-EC2, minus some of the nice trappings like load balancers.

At $DAYJOB, we use the Telmate terraform proxmox provider, too. Upgrades over the years have had a few papercuts (mainly when values are shifted around), but it has overall been an immense timesaver. A terraform repo contains the 2 colocation datacenter proxmox environments plus a local lab edition of the same builds (locallab uses containers vs full VM's on the "real" hardware so that I can run almost all of the configuration on my laptop). Saltstack gets deployed during the buildouts, which picks up the OS/ application configuration after the OS is deployed.

No agentic stuff on our stack, as our security posture can't afford that currently.

As for the load-balancing, I think the later versions have supported targeting proxmox clusters vs a single node, and the newest Proxmox can do resource auto-balancing. That might get you what you need

Re: Migrating from Proxmox to NixOS and Incus

#42

Post by whalesalad »

h4kunamata wrote: post #37
whalesalad wrote: post #12

I haven't abandoned Proxmox yet, but the take here resonates with me. I do not like configuring appliances. I prefer defining infra as code, having that diffable, assertable, etc. I have had pretty good luck managing Proxmox clusters with the Proxmox API ( https://pve.proxmox.com/pve-docs/api-viewer/index.html ) or just letting the agent shell in as root (lol). I built a very simple provisioning tool called vmfactory…

>I do not like configuring appliances. I prefer defining infra as code, having that diffable, assertable, etc. I hardle ever access Proxmox GUI, everything is fully managed via Ansible playbooks, from deploying the LXC containers to updating them, from taking snapshot to syncing stuff to my TrueNAS is done via batching. >I did abandon TrueNAS, however. It really is a locked-down appliance. Good luck installing custom…

[dead]

Re: Migrating from Proxmox to NixOS and Incus

#43

Post by lmm »

h4kunamata wrote: post #31

>By moving to NixOS, my entire infrastructure is defined in text files. This means my AI agents can read, understand, and even safely modify my infrastructure You lost me there!! Firstly, NixOS is hype, like everything being moved to Rust and failing miserably. Secondly, "AI ... can .... safely modify my infrastructure", OP is either being a troll or haven't seem how the whole IT world is upside down because of those…

> Secondly, "AI ... can .... safely modify my infrastructure", OP is either being a troll or haven't seem how the whole IT world is upside down because of those very same statements.

That's exactly why you want your infrastructure defined in version controlled files with easy revert when something gets screwed up.

> Thirdly, "my entire infrastructure is defined in text files", you clearly never heard of Ansible.

Ansible is a half-assed version of what nix or even Puppet gets you. Having a version controlled record of which shell commands you ran doesn't help that much when you're still running random uncontrolled shell commands and hoping they do what you want them to.

Re: Migrating from Proxmox to NixOS and Incus

#44

Post by EnigmaCurry »

dereknance wrote: post #13
EnigmaCurry wrote: post #4

NixOS has transformed my use of Proxmox. I configure, build, and deploy everything from my nix workstation. I don't need to use the PVE gui at all. Proxmox is just a target, and I've abstracted things enough to where I can deploy the same machines to libvirt on a local machine too. Why would I need to let my agent into my PVE box? I haven't looked at incus, but if I wanted to run the full stack declaratively, nixos a…

> I configure, build, and deploy everything from my nix workstation. I don't need to use the PVE gui at all. I would love to know more about how you do this, particularly the deploy part. I'm considering moving away from Ansible, but haven't had the time to dedicate to exploring a similar Nix experience.

This is what I've built [1], and I'm using it daily for things like agent sandboxing when I deploy it builds a new image completely replaces the OS disk - on upgrade it only keeps its data drive mounted to /var.

[1] https://github.com/EnigmaCurry/nixos-vm-template

Re: Migrating from Proxmox to NixOS and Incus

#45

Post by EnigmaCurry »

daishi55 wrote: post #9
EnigmaCurry wrote: post #4

NixOS has transformed my use of Proxmox. I configure, build, and deploy everything from my nix workstation. I don't need to use the PVE gui at all. Proxmox is just a target, and I've abstracted things enough to where I can deploy the same machines to libvirt on a local machine too. Why would I need to let my agent into my PVE box? I haven't looked at incus, but if I wanted to run the full stack declaratively, nixos a…

What do you mean exactly? You have a nix workstation and a physically separate PVE server, and configure the second from the first?

Yes, I don't really touch the PVE server at all after the initial install. I just build and deploy from my Fedora workstation running nix.

[1] https://github.com/EnigmaCurry/nixos-vm-template

Re: Migrating from Proxmox to NixOS and Incus

#48

Post by catdog »

mikestorrent wrote: post #20
whalesalad wrote: post #12

I haven't abandoned Proxmox yet, but the take here resonates with me. I do not like configuring appliances. I prefer defining infra as code, having that diffable, assertable, etc. I have had pretty good luck managing Proxmox clusters with the Proxmox API ( https://pve.proxmox.com/pve-docs/api-viewer/index.html ) or just letting the agent shell in as root (lol). I built a very simple provisioning tool called vmfactory…

https://github.com/Telmate/terraform-provider-proxmox ... I run Ansible against Proxmox hosts to ensure the config on them is what I want, and then Terraform all VMs into place on them. It's not too far off from having your own mini-EC2, minus some of the nice trappings like load balancers.

Switched to that one some time ago, more features and adapts faster to changes in newer proxmox versions https://github.com/bpg/terraform-provider-proxmox

Re: Migrating from Proxmox to NixOS and Incus

#49

Post by daft_pink »

I find I have a lot of trouble with updating with nixos. I really like the OS but I find that when I have something like Claude Code and I need to update to the newest version, I just have trouble. It just seems like a lot of work updating versions otherwise NixOS is really awesome and great.

Re: Migrating from Proxmox to NixOS and Incus

#50

Post by itchingsphynx »

sfRattan wrote: post #21

Another proxmox-esque project I've been watching is Sylve , a control plane for FreeBSD that provides a web interface to jails, bhyve VMs, and containers [1]. It's new-ish, but it looks like a possible sweet spot replacement for both proxmox and TrueNAS (which was originally also built on FreeBSD before they switched to Linux IIRC), at least for my homelab-ing use case. Potentially eventually for environments at grea…

How does Sylve compare to Webmin [1]?

[1]: https://docs.vultr.com/how-to-install-webmin-on-freebsd-14-0

Post reply on HN