Earlier quoted context omitted.
The Qubes people don't recommend doing GPU passthrough because of the security implications. As for the OP, I feel like if somebody cares about security, they shouldn't be doing any of this. Trying to come up with some self-designed hodgepodge of things isn't really enough security-wise, even if you do use VMs, and I'd find it hard to trust something like this as a platform to do anything important on.
> The Qubes people don't recommend doing GPU passthrough because of the security implications. Why? DMA?
See also: https://www.qubes-os.org/faq/#can-i-run-applications-like-ga...
But the fact is, even if you are doing GPI passthrough in Qubes, it's much more secure than running any other system.