I really wish we could name individual devices so we can target Siri request to specific devices. Partially for situations like you describe.
Also not all siri+device combos can do the same things. Our HomePod is not customized so anyone can use it for basic things like playing music but then it can't do personalized tasks like set reminders. Some apps only exist on some devices and a Siri request involving those will fail on other devices.
I've turned off "hey Siri" for most devices and only use it for the HomePods who really need it to be useful. For my phone or watch, I hold down the button and the whisper to the device hoping the HomePod down the hall doesn't hear and start playing something in the living room I only wanted to play in my office.