Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Proxmox virtual environment 9.1 available

proxmox.com

Post reply on HN
Print view View original

71–80 of 81 posts

Re: Proxmox virtual environment 9.1 available

#71

Post by tlamponi »

nyrikki wrote: post #70
tlamponi wrote: post #67

Earlier quoted context omitted.

Correction: In Proxmox VE we're not using virsh/libvirt at all, rather we have our own stack for driving QEMU on a low-level, our in-depth integration, especially with live local storage migration our Backup Servers dirty-bitmap (known as change block tracking in vmware worlds) would be possible in the form we have it. Same w.r.t. our own stack for managing LXC container. The web UI part is actually one of our smalle…

Correct sorry I don't use the web-ui's and was confusing oVirt, I forgot that you are using perl modules to call qemu/lxc. I would strongly suggest more work on your NUMA/cpuset limitations. I know people have been working on it slowly but with the rise of E and P cores, you can't stick to pinning for many use cases and while I get hyperconvergence has it's costs, and platforms have to choose simple, the kernels cpus…

PVE itself is still made of a lot of perl, but nowadays, we actually do almost everything new in rust.

We already support CPUsets and pinning for Container VMs, but definitively can be improved, especially if you mean something more automated/guided by the PVE stack.

If you have something more specific, ideally somewhat actionable, it would be great if you could create an enhancement request at https://bugzilla.proxmox.com/ so that we can actually keep track of these requests.

Re: Proxmox virtual environment 9.1 available

#72

Post by Havoc »

hendersoon wrote: post #11

So with support for OCI container images, does this mean I can run docker images as LXCs natively in proxmox? I guess it's an entirely manual process, no mature orchestration like portainer or even docker-compose, no easy upgrades, manually setting up bind mounts, etc. It would be a nice first step.

There is a vid showing the process on their youtube

https://youtu.be/4-u4x9L6k1s?t=21

>no mature orchestration

Seems to borrow the LXC tooling...which has a decent command line tool at least. You could in theory automate against that.

Presumably it'll mature

Re: Proxmox virtual environment 9.1 available

#73

Post by m463 »

summermusic wrote: post #51
jmward01 wrote: post #4

Watching hypervisors slowly improve over the last few years has been amazing. They aren't quite to the point that I will install them under any new hardware I buy and then put my daily driver OS on top, but they are very close. I think a strong focus on creating 'the OS under your OS' experience seamless could open up a lot more here.

I have a PC where I installed Proxmox on bare metal and put a daily-use desktop OS on top. It works surprisingly well, the trickiest part was making sure the desktop OS took control of video/audio/peripherals.

I always thought it might be cool to be able to do this with a laptop.

I thought it might need gpu virtualization?

do you do it with passthrough?

Re: Proxmox virtual environment 9.1 available

#74

Post by m463 »

I've always just clean-installed then copied the containers/vms using vzdump and pct restore/qmrestore

I learned stuff like this years ago with upgrades to debian/ubuntu/etc - upgrading a distribution is a mess, and I've learned not to trust it.

Re: Proxmox virtual environment 9.1 available

#76

Post by rcarmo »

veidr wrote: post #3

Nah. Incus. Sorry, but I bought Proxmox 7, but it is not comparable. Incus does everything (and more) with better interface, WAY better reliability, and also not like a hundred EUR or whatever. (100 EUR is fine with me if better, but not if not better...)

Incus has a pretty bare-bones web UI without nary any metrics, which is a bit of a pain when you're trying to track down CPU hogs. Proxmox, on the other hand, makes things very much visible and easy to use.

Re: Proxmox virtual environment 9.1 available

#77

Post by fuzzylightbulb »

zamadatix wrote: post #35
throw0101c wrote: post #2

Proxmox (and XCP-ng?) seems to be "the" (?) popular alternative to VMware after Broadcom's private equity-fuel cash grab. (Perhaps if you're a Microsoft shop you're looking at Hyper-V?)

Nutanix is popular with traditional larger enterprise VMware type customers, Proxmox is popular with the smaller or homelabber refugees. Exceptions exist to each of course.

That people consolidated their business atop VMware's hypervisor, got screwed by Broadcom, and as a result are moving everything to Nutanix (from whom they need to buy the hypervisor, the compute stack, the storage stack, etc.) is insane to me.

Re: Proxmox virtual environment 9.1 available

#78

Post by nyrikki »

tlamponi wrote: post #71
nyrikki wrote: post #70

Earlier quoted context omitted.

Correct sorry I don't use the web-ui's and was confusing oVirt, I forgot that you are using perl modules to call qemu/lxc. I would strongly suggest more work on your NUMA/cpuset limitations. I know people have been working on it slowly but with the rise of E and P cores, you can't stick to pinning for many use cases and while I get hyperconvergence has it's costs, and platforms have to choose simple, the kernels cpus…

PVE itself is still made of a lot of perl, but nowadays, we actually do almost everything new in rust. We already support CPUsets and pinning for Container VMs, but definitively can be improved, especially if you mean something more automated/guided by the PVE stack. If you have something more specific, ideally somewhat actionable, it would be great if you could create an enhancement request at https://bugzilla.proxm…

There is a bit of a problem with polysemy here.

While the input for qemu is called a "pve-cpuset" for affinity[0], it is using explicitly the taskset[1][3] command.

This is different than cpuset[2], or how libvirt allows the creation of partitions[3] using systemd slices in your case.

The huge advantage is that setting up basic slices can be done when provisioning the hypervisor, and you don't have the hard code cpu pinning numbers as you would in taskset, plus in theory it could be dynamic.

From the libvirt page[4]

     ...
     
       /machine/production
     
     ...
As cpusets are hierarchical, one could use various namespace schemes, which change per hypervisor, not exposing that implementation detail to the guest configuration. Think migrating from an old 16 core CPU to something more modern, and how all those guests will be pinned to a fraction of the new cores without user interaction.

Unfortunately I am deep into podman right now and don't have a proxmox at the moment or I would try to submit a bug.

This page[5] covers how even inter CCD traffic even on Ryzen is ~5x compared to local. That is something that would break the normal affinity if you move to a chip with more cores on a CCD as an example. And you can't see CCD placement in the normal numa-ish tools.

To be honest most of what I do wouldn't generalize, but you could use cpusets, with a hierarchy and open the choice to try and improve latency without requiring each person launching a self service VM to hard code the core ID's.

I do wish I had the time and resources to document this well, but hopefully that helps explain more about at least the cpuset part, not even applying the hard partitioning you could do to ensure say ceph is still running when you start to thrash etc...

[0] https://git.proxmox.com/?p=qemu-server.git;a=blob;f=src/PVE/...

[1] https://git.proxmox.com/?p=qemu-server.git;a=blob;f=src/PVE/...

[2] https://docs.kernel.org/admin-guide/cgroup-v2.html#cpuset

[3] https://man7.org/linux/man-pages/man1/taskset.1.html

[4] https://libvirt.org/cgroups.html#using-custom-partitions

[5] https://kb.blockbridge.com/technote/proxmox-tuning-low-laten...

Re: Proxmox virtual environment 9.1 available

#79

Post by zamadatix »

fuzzylightbulb wrote: post #77
zamadatix wrote: post #35

Earlier quoted context omitted.

Nutanix is popular with traditional larger enterprise VMware type customers, Proxmox is popular with the smaller or homelabber refugees. Exceptions exist to each of course.

That people consolidated their business atop VMware's hypervisor, got screwed by Broadcom, and as a result are moving everything to Nutanix (from whom they need to buy the hypervisor, the compute stack, the storage stack, etc.) is insane to me.

Most don't even consider the amounts as getting screwed, just enough change that on the next refresh cycle it was worth switching to a different provider. For a lot of these places it was just 10-15 years ago they went from 0 VMs to 80%+ VMs so they aren't worried about needing to move around, just the quality of the support contract etc.

Re: Proxmox virtual environment 9.1 available

#80

Post by gigel82 »

tlamponi wrote: post #68
gigel82 wrote: post #33

What is this "application containers" BS, just add native docker stack support. Most folks in the self hosting community already deploy nested dockers in LXCs, just add native support so we can cut out the middle man and squeeze out that indirection.

It makes no sense to add an extra layer, and we definitively do not want to make us and our users dependent of docker project. There exist many OCI runtimes, and our container toolkit already provides a (ball parked) 90% feature overlap with them. Maintaining two stacks here is just needless extra work and asking for extra pain for us devs and our users, so no, thanks. That said, PVE is not OCI runtime compatible yet…

People stuck with Docker for a reason, even after they became user hostile. Almost every selfhosted project in existence provides a docker-compose.yml that's easy to expand and configure to immediately get started. None provide generic OCI containers to run in generic OCI runtimes.

I understand sticking with compatibility at that layer from an "ideal goal" POV, but that is unlikely to see a lot of adoption precisely because applications don't target generic OCI runtimes.

Post reply on HN