And I don't think it'll get better, with Sonoma I'm seeing more and more audio switching bugs and weird behaviours when I connect headphones. At least this balance bug disappeared, although it was plagueing me on my previous MacBook.
I am currently experiencing an unbelievably annoying bug where once I finish on Google Meet in Firefox, all audio/video across the whole OS plays in slo-mo. I have to turn my headphones off and back on to fix it. It's something specific to these headphones (Sony WH-C700N), as they do something different to get higher audio quality when using the microphone. Apple have helpefully removed Bluetooth Explorer from Xcode…
Sounds like a sample rate inconsistency issue, maybe a trip to Audio MIDI Setup.app could give hints.