Just use NVIDIA Broadcast. Filters out everything but talking.
NVIDIA Broadcast is only available on Windows (but free and works very well if you have a RTX card). Use it if available! If using Discord, Krisp is built-in. On Windows/Mac, you can also install Krisp standalone. On Linux, I would recommend NoiseTorch or other similar software.
If you have a non-RTX nVidia GPU, you can use RTX Voice, which is the same functionality as the nVidia Broadcast noise removal, and doesn't require an RTX card despite the name.
I was using RTX Voice on my GTX 1070 just fine.