Earlier quoted context omitted.
Why do you need to use a KVM rather than Sunshine+Moonlight? That's pretty much the gold standard for remote gaming.
The practical answer for right now is that I don't want software screen capture, but it also does not do any of the things I wanted to be able to do by having remote inputs directed to the hardware rather than simulated in software. Namely it requires software installation on the server, which means OS support becomes a thing, you can't use it in BIOS/firmware, etc.
Note that if anti-cheat is a concern, common dev boards like Arduino Leonardo and Pi Pico could also trigger flagging, so maybe that needs to be done slightly differently too(assuming the intent is good and this don't count towards anti-cheat evading in whatever you're doing).