Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Migrating from Proxmox to NixOS and Incus

nijho.lt

Post reply on HN
Print view View original

21–30 of 73 posts

Re: Migrating from Proxmox to NixOS and Incus

#21

Post by sfRattan »

Another proxmox-esque project I've been watching is Sylve, a control plane for FreeBSD that provides a web interface to jails, bhyve VMs, and containers [1]. It's new-ish, but it looks like a possible sweet spot replacement for both proxmox and TrueNAS (which was originally also built on FreeBSD before they switched to Linux IIRC), at least for my homelab-ing use case. Potentially eventually for environments at greater scale also. The company behind it is a software consultancy and Sylve is built with their actual business needs in mind, and is BSD-licensed like the OS it runs atop.

One of the developers building Sylve gave a talk last year [2].

[1]: https://sylve.io/

[2]: https://youtu.be/wo4oD5UON30

Re: Migrating from Proxmox to NixOS and Incus

#22

Post by scorpioxy »

yobert wrote: post #15
gchamonlive wrote: post #10

Earlier quoted context omitted.

Incus is roughly a frontend for qemu, so you can launch an empty VM and use the ISO to install the OS. You don't have to use a preconfigured base image.

I think incus can be a frontend for qemu, but it's primary mode of operating is to run containers. It's a fork of LXD.

VM support has been there for quite a while now and works nicely. I think that's what they're referring to. It started out with only system containers and then gained support for VMs and now there's work to support launching OCI images directly.

Re: Migrating from Proxmox to NixOS and Incus

#23

Post by scorpioxy »

sbstp wrote: post #11

Incus is great. I've been trying to revive an unmaintained ansible collection to manage incus resources https://github.com/sbstp/ansible-collection-incus

I don't if you know this, but there was a collection for LXD. Last time I checked, it didn't seem very popular so not maintained very well but it did work. Maybe that could be used for inspiration.

I remember Stéphane worked on adding support for incus containers to opentofu which seemed more popular than using ansible to describe the resources.

Re: Migrating from Proxmox to NixOS and Incus

#24

Post by sieabahlpark »

mikestorrent wrote: post #18
redlewel wrote: post #17

I get the advantage of fully declarative systems such as Nix, but everything you do in proxmox via GUI can be done via CLI: https://pve.proxmox.com/pve-docs/#_command_line_interface CLI is first class in proxmox, I use the qm command for managing vms all the time. The networking is also just a file in `/etc/network/interfaces` that I modify with vim as needed.

CLI is dandy but Terraform is even nicer. I run something like ten Proxmox clusters now around the world, and an AI agent can deploy a fully terraformed VM for me, tied into monitoring, asset mgmt, EDR, etc in a few minutes...

[dead]

Re: Migrating from Proxmox to NixOS and Incus

#25

Post by RossBencina »

redlewel wrote: post #17

I get the advantage of fully declarative systems such as Nix, but everything you do in proxmox via GUI can be done via CLI: https://pve.proxmox.com/pve-docs/#_command_line_interface CLI is first class in proxmox, I use the qm command for managing vms all the time. The networking is also just a file in `/etc/network/interfaces` that I modify with vim as needed.

Doing things via the CLI and editing config files doesn't automatically give you reproducability or audit trail. Last time I check etckeeper doesn't work with proxmox either (due to the proxmox FUSE mount in /etc).

Re: Migrating from Proxmox to NixOS and Incus

#26

Post by [deleted] »

kennywinker wrote: post #5

> But fundamentally, Proxmox is built around clicking buttons. It is a GUI-first paradigm. Uhh, whut? It provides a button-y interface, but you can do everything via config files and `pct` on the command line if you prefer. I know that’s not full nix-style declarative, but you don’t have to mislead to sell me on the advantages of declarative infra.

[deleted]

Re: Migrating from Proxmox to NixOS and Incus

#27

Post by linsomniac »

It's really hard to tell what the benefits of Incus over Ganeti are. The Incus site doesn't really lead with features or benefits that I can see. About the only things I could figure out are that Incus can manage containers and can manage instance creation through cloning a base volume, where ganeti (afaik) always creates volume and populates it. Well, and Incus seems to be MUCH more popular than Ganeti.

I searched the documentation but it wasn't really clear what its live migration and ZFS migration story is, but when I asked Claude to research it, it tells me that it supports live migration via ZFS snapshot replication, which is exactly what I'm looking for. I implemented a ganeti storage driver that does the same thing and am just getting ready to start testing it, but if Incus supports it I might look at moving that direction.

Anyone use Incus live migration with ZFS?

Re: Migrating from Proxmox to NixOS and Incus

#29

Post by linsomniac »

>The Agentic Multiplier

Oh man, you buried the lede there.

I switched over to NixOS around a month ago from Ubuntu and it's been just a dream. I expected there to be some friction with installing things that aren't already built for NixOS, but honestly it's been easier with LLM+NixOS than it was with Ubuntu.

edit: Thinko

Re: Migrating from Proxmox to NixOS and Incus

#30

Post by evanjrowley »

I've recently migrated to Incus and am enjoying it so far. Hopefully the good feelings last.

Roughly ten years ago, my homelab consisted of a dozen virtual machines running on SmartOS. I was not familiar with Illumos, and this was before it had a widely available web UI, but it was simple enough to use that these challenges didn't matter much. SmartOS was designed to boot reliably from USB flash storage, allowed me to use all my SATA ports for VM storage, and was my first "immutable" operating system. The primary focus on ZFS storage was another great quality of SmartOS.

Two moves and several years later, it was time to rebuild the lab, and I decided to go with Proxmox because it had decent ZFS support. Experience with Proxmox has been very good too. The GUI, many more virtualization features (in addition to the key ones I care about), and better hardware support through the Linux kernel have kept me on Proxmox for a long time.

Customizing my Proxmox installation always gave me anxiety. How could I defend my hypervisor from configuration drift? I wished there could be an immutable version of Proxmox.

Later on, I learned about govulcheck, which offers a novel dynamic/static analysis hybrid approach to vulnerability management. Nothing else out there does this (without teaming up with some huge company). I began to think that I should favor software solutions based on golang.

Ultimately, Incus (and IncusOS) fit this need very well. My IncusOS hosts excellent and I'm glad I can run Incus itself on most Linux distros - including NixOS!

I'll keep a small Proxmox host around for experimenting with new kernel features (Intel GVT-g / SR-IOV graphics) and old operating systems like Windows XP or anything else that needs special QEMU options.

Post reply on HN