Earlier quoted context omitted.
The easiest solution is to just buy an HDMI switch(there's loads of them, even ones which work with 4K signal, and they don't disrupt DHCP) and a USB switch. Total solution should fit in about 40-50 dollars.
But, that's more than one button to push! /s Seriously though, I am a regular user of DVI and VGA KVMs, which have served me well over the years, but now as I migrate to HDMI or DisplayPort connectors and higher resolutions, I'm finding myself having to re-think how I hook everything together.
I tried plugging the cable into my KVM switch and it worked okay on the Windows 10 side but there was some hot-plugging kernel panic when switching to Linux (must file a bug report!)
So perhaps the future of KVM switches doesn't need video connectors such as VGA/DVI/DP if USB-C -> HDMI (as smartphones are adopting) becomes universal.