Earlier quoted context omitted.
Not as secure as VMs but GPU passthrough with Docker/Podman is much easier to set up, and you can even use the GPU on the host machine at the same time.
Are you giving it access to /dev/dri, or doing some fancier sandboxing? (Would you even need anything fancier? I think /dev/dri is supposed to isolate users.)
[1] https://docs.nvidia.com/datacenter/cloud-native/container-to...