I've created around 400+ video tutorials with Camtasia, it works well enough and IMO is one of the best all-in-1 screencast recording and editing tools you'll find if you're on Windows or MacOS. Lately I've been using OBS for recording because I want to use native Linux and Camtasia does not run on Linux. Using a Windows / Mac VM won't work if you want to record your Linux environment. Then I use DaVanci Resolve to d…
Have you tried HDMI to USB video capture? If you computer has enough power you may pass-thru USB3 and audio into VirtualBox with Windows guest running Camtasia. To avoid extra hardware maybe even VNC would be enough. That is just a hack idea and I am not Camtasia user yet.
I've looked into using a 2nd GPU and doing a pass-through with KVM which would let you run a Windows VM with effectively native performance. I'm sure that would be great for editing, but it's the recording the Linux environment outside of the VM that would be the issue. Audio is super sensitive with latency, even as little as 15ms of latency could be enough to kill an audio recording.
Needless to say that's why I'm striving to get DaVinci to work for the editing side of things. It would be so nice to avoid having to do any of the above and just use native Linux straight up with no VMs or secondary machine.