In Shared NVSwitch Multitenancy Mode - are there any considerations for leveraging infiniband devices inside each vm at full performance?
Virtualizing Nvidia HGX B200 GPUs with Open Source
11–20 of 32 posts
Re: Virtualizing Nvidia HGX B200 GPUs with Open Source
#12(author of the blog post here) For me, the hardest part was virtualizing GPUs with NVLink in the mix. It complicates isolation while trying to preserve performance. AMA if you want to dig into any of the details.
Isn't SR-IOV a thing with these big GPUs? Or, is it that you're not concerned with fractional granularity?
Re: Virtualizing Nvidia HGX B200 GPUs with Open Source
#13Is Nvidia’s Fabric Manager and other control plane software Open Source? If so, that’s news to me. It’s not clear that anything in this article relates to Open Source at all; publishing how to do VM management doesn’t qualify. Maybe “open kimono.” Also, how strong are the security boundaries among multiple tenants when configured in this way? I know, for example, that AWS is extremely careful about how hardware resou…
Fabric Manager itself is not open source. It's NVIDIA-provided software, and today it's required to bring up and manage the NVLink/NVSwitch fabric on HGX systems. What we meant by "open" is that everything around it - the hypervisor, our control plane logic, partition selection, host configuration, etc. - is implemented in the open and available in our repos. You're right that this isn't a fully open GPU stack. On is…
[deleted]
Re: Virtualizing Nvidia HGX B200 GPUs with Open Source
#14Did you ever manage to get vGPU's working in any other hardware configuration? I know it's not what Hx00 customers want. I bloodied my forehead on that for a month or two with Cloud Hypervisor --- I got to the "light reverse engineering of drivers" stage before walking away.
Re: Virtualizing Nvidia HGX B200 GPUs with Open Source
#15Cool post. Have you looked at slicing a single GPU up for multiple VMs? Is there anything other than MIG that you have come across to partition SMs and memory bandwidth within a single GPU?
Last I checked MIG was the only one that made hard promises about especially memory bandwidth; as long as your memory access patterns aren't secret and you have enough trust in the other guests not being highly unfriendly with their cache usage behavior, you should be able to get away with much less strict isolation. Think docker vs. VMs-with-dedicated-cores. But I thought MIG did do the job of chopping a GPU that's…
Can you pass a MIG device into a KVM VM? The team we worked with didn't believe it was possible (they suggested we switch to VMWare); the MIG system interface gives you a UUID, not a PCI BDF.
Re: Virtualizing Nvidia HGX B200 GPUs with Open Source
#16In Shared NVSwitch Multitenancy Mode - are there any considerations for leveraging infiniband devices inside each vm at full performance?
We haven't looked deeply at inter-machine communication yet. NVLink/NVSwitch (which this post focuses on) are intra-node, so InfiniBand is mostly orthogonal I think and comes down to NIC passthrough, NUMA/PCIe placement, and validating RDMA inside the VM.
Re: Virtualizing Nvidia HGX B200 GPUs with Open Source
#17(author of the blog post here) For me, the hardest part was virtualizing GPUs with NVLink in the mix. It complicates isolation while trying to preserve performance. AMA if you want to dig into any of the details.
Isn't SR-IOV a thing with these big GPUs? Or, is it that you're not concerned with fractional granularity?
In this article, we're primarily concerned with whole-GPU or multi-GPU partitions that preserve NVLink bandwidth, rather than finer-grained fractional sharing of a single GPU.
Re: Virtualizing Nvidia HGX B200 GPUs with Open Source
#18Did you ever manage to get vGPU's working in any other hardware configuration? I know it's not what Hx00 customers want. I bloodied my forehead on that for a month or two with Cloud Hypervisor --- I got to the "light reverse engineering of drivers" stage before walking away.
We didn't focus on vGPU and largely avoided it on purpose. Instead, we focused on whole-GPU and NVSwitch-partitioned passthrough (Shared NVSwitch Multitenancy Mode), which is a better fit for the workloads we care about.
Re: Virtualizing Nvidia HGX B200 GPUs with Open Source
#19can someone explain me like I'm 10 what is a BAR?
Like it says something about mmaping 256 GB of per GPU. But wouldn't it waste 2T of RAM? or do I fail in my understanding of what "mmap" is as well..
EDIT: yes, seems like my understanding of mmap wasn't good, it wastes not RAM but address space
Re: Virtualizing Nvidia HGX B200 GPUs with Open Source
#20A lot of this coincides with my own experiments I did to pass-through consumer AMD GPUs into VMs [1], which the Debian ROCm Team uses in their CI.
The Debian package rocm-qemu-support ships scripts that facilitate most of this. I've since generalized this by adding NVIDIA support, but I haven't uploaded the new gpuisol-qemu package [2] to the official Archive yet. It still needs some polishing.
Just dumping this here, to add more references (especially the further reading section, the Gentoo and Arch wikis had a lot of helpful data).
[1]: https://salsa.debian.org/rocm-team/community/team-project/-/...