In my experience the whole VFIO subsystem is quite stable and at least Nvidia hardware can be detached and attached to a different driver without any difficulty. Intel GPUs have trouble being reattached to the host because of driver issues and AMD needs workarounds because of hardware level bugs. Also, if you have a laptop, or a desktop Intel, you already have a perfectly usable GPU that the host can use included in the CPU!
It's true that setting up everything can take some time to get it working, but it depends very much on the hardware. In most cases on a desktop system all you need to do is select the right PCI device in libvirt's GUI and it should work. On laptops you need to hack together a custom OVMF UEFI firmware imagine with the VBIOS included, but there are ways to auto-generate it [1].
Input can be handled in multiple ways, which makes most sense depends on if you want to use dedicated output, or if you want to stream the output back to the host using something like Looking Glass. I just have a virtual qxl GPU in the VM which lets me use the qemu window for input and that works fine.
For audio there is scream[2], which gives me super-low latency 7.1 audio. According to pulseaudio latency is under 2ms in addition to whatever latency Windows itself has. I use a -ck kernel with MuQSS scheduler, the default scheduler won't be able to go that low.
I mean it can be tricky to set up, but if the hardware is capable it shouldn't be more than a weekend project. Once set up it's rock solid stability wise and compatibility is the same as bare metal. Caveat is some online games consider running in a VM cheating, but that doesn't affect me. It's not for everyone, but neither is Linux as a desktop OS
Overall I consider this the holy grail for those that want to primarily run Linux but don't want to give up games or deal with wine/proton issues. Actual dual booting is completely obsolete with GPU passthrough in my opinion. YMMV, depending on what you play and your exact hardware.
[1]: https://github.com/bitthief/ovmf-with-vbios-patch
[2]: https://github.com/duncanthrax/scream