Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running MicroVMs in Proxmox VE, the Easy Way

taoofmac.com

Post reply on HN
Print view View original

31–40 of 52 posts

Re: Running MicroVMs in Proxmox VE, the Easy Way

#31

Post by pm »

alexellisuk wrote: post #8

This is clever work, especially given that Proxmox is already a very viable VMware replacement and wasn’t originally designed around microVMs as the primary abstraction. I’m glad this is working well for you. We’ve been on a similar journey, but came at it from the opposite direction. We started SlicerVM in 2022 after seeing how slow Multipass felt when launching more than one Linux VM, even though it is relatively l…

Hey Alex! Just wanted to say I came across Slicer three or so months ago, researching automatic provisioning for RPis (and other devices in the homelab). It piqued my interest, as I was also researching solutions for partitioning and distributing workloads. I hope to give Slicer a go within the next quarter, however I'm still getting to grips with the fundamentals of distributed computing. Either way, it's interesting work.

Re: Running MicroVMs in Proxmox VE, the Easy Way

#32

Post by pm »

binsquare wrote: post #24
traceroute66 wrote: post #11

Earlier quoted context omitted.

> We started SlicerVM .... Shame you did not mention once in your long post that you are based on Firecracker, because I'm sure I'm not the first who was about to post "why is this better than Firecracker". Also it is a shame you've adopted the subscription billing model instead of allowing people to buy perpetual licenses. I dislike the subscription model in a pure sense, but also I dislike the "but its 'only' $x a…

Author of a free open source alternative to slicer, based on a fork of libkrun (not firecracker based) that runs locally across wsl, macOS, and Linux natively. https://github.com/smol-machines/smolvm

I noticed this when it was first announced on HN as well. Hoping to give it a try as well shortly.

Re: Running MicroVMs in Proxmox VE, the Easy Way

#34

Post by sgc »

FireBeyond wrote: post #22
tlamponi wrote: post #5

FWWI, we did evaluate and benchmark microVMs back in 2020. Back then it was not really seen worth it the maintenance cost compared to what it brought to the table, but it makes sense to re-evaluate that again soonish; with native dynamic load balancing and affinity rules (and further orchestration improvements being lined up) they might be better leveraged today. Oh, and mailing lists are a bliss to use compared to (…

Yeah, I love this from a geek appeal and I have a beefy home lab (Dual Xeon Silver 4116 with 384GB, and 12TB of RAID10 SSD connected via 10gig), but being a homelab I want to eke out all my performance so I keep looking at LXC on my Proxmox box versus the optimization of VMs (does this machine really need 8GB or 6? Or 5?). But when there's the discussion of the amount of time Qemu spends "in grub" and "probing legacy…

I think in most scenarios you don't need to worry so much about kvm ram use, since it looks static but actually it's not and you can over-commit [1]. And of course disk allocation can be dynamic as well. I prefer a lot more security for a bit less flexibility. I am not as ram rich as you are, and still every time I think of my few LXCs, my main thought is 'why did I do that?'.

[1] https://docs.redhat.com/en/documentation/red_hat_enterprise_...

Re: Running MicroVMs in Proxmox VE, the Easy Way

#35

Post by INTPenis »

j45 wrote: post #23
agentifysh wrote: post #21

proxmox has been great although it comes with a learning curve back when I used to use cursor I build this mcp but it should work for codex or claude it lets me easily spin up vms with specs its tough to create boxes now due to ram prices but got mine at a great time when it was very cheap; i just wish i had bought more then https://github.com/agentify-sh/cursor-proxmox-mcp

Can you share more about the learning curve specifics you're referring to? Thanks.

I'm not the person you replied to, but I came from using VMware products for 12 years, to using Proxmox this last 1.5 years.

These are my impressions.

First of all it's a very competent product, mainly thanks to Ceph making it HCI. Without Ceph, I'm not sure what we would do.

It's as effective as you design it, make sure to separate storage and cluster traffic to ensure robustness, and speed. Make sure to use at least 10GbE switch for storage, for fast migrations.

And managing ceph is very important, basically boils down to 1) never let it run out of space, and 2) the more devices you have the easier it is to manage.

Automating against Proxmox definitely is the biggest pain point, and this needs the most work done.

I've spent countless hours, pre-AI, building our automation setup using both Terraform and Ansible. I sort of wish I had tried AI earlier because it does make things easier.

Some things like automating the creation of templates will forever be a complex procedure in Ansible. And I abandoned Terraform completely because the API was too unpredictable for Terraforms strict state, Ansible was a much better fit.

Their AuthZ takes some getting used to, the fact that if you select "Privilege Separation" it countes the user's permissions AND the token permissions, and the token permissions must always be lower than the users.

Templates existing on one node, but taking a unique VM ID across the cluster is also a bit confusing. It means in practice we're always deploying VMs on the same node, before migrating them somewhere else.

Re: Running MicroVMs in Proxmox VE, the Easy Way

#36

Post by dizhn »

sorenjan wrote: post #17

Am I understanding the ballooning part right that this doesn't allocate all of the VM memory from the host until it's needed, and releases memory automatically when it's not needed anymore? So you can overprovision memory with multiple guests as long as the guests aren't using the memory at the same time?

That's true but there are peculiarities. People usually don't do it on Windows VMs for instance. I myself do not use it at all but prox also does KSM (Kernel Samepage Merging) which activates when RAM usage is at a particular configurable level and helps a lot more than ballooning especially if your VMs and internals are similar.

Re: Running MicroVMs in Proxmox VE, the Easy Way

#37

Post by dizhn »

j45 wrote: post #25

I'm curious if there's more than one way to achieve this. Creating a single VM, with vm within vm (performance hit would be negligible for the orchestration work of agents), and it might offer some alternatives without having to customize Proxmox as much?

Up until the final release that's exactly how they recommended using docker so, yes.

Re: Running MicroVMs in Proxmox VE, the Easy Way

#38

Post by dizhn »

LorenDB wrote: post #10

One of my Proxmox hosts is glacially slow at running VMs. (Dell R520; I have a same-generation server that is fine at VMs, so not sure what the root cause is). I wonder if this would help performance.

Doubt it because you clearly have a problem there. Same kind of disk?

Re: Running MicroVMs in Proxmox VE, the Easy Way

#39

Post by KetoManx64 »

INTPenis wrote: post #35
j45 wrote: post #23

Earlier quoted context omitted.

Can you share more about the learning curve specifics you're referring to? Thanks.

I'm not the person you replied to, but I came from using VMware products for 12 years, to using Proxmox this last 1.5 years. These are my impressions. First of all it's a very competent product, mainly thanks to Ceph making it HCI. Without Ceph, I'm not sure what we would do. It's as effective as you design it, make sure to separate storage and cluster traffic to ensure robustness, and speed. Make sure to use at leas…

When it comes to templates, I shut then down and take a backup, and then just restore that backup to a different node
Post reply on HN