Agggrrrr so much hate for these devices. I bought a Mini for my oldest kid and now, when I pick up my phone in my downstairs office, hold it near my mouth, and say "hey Siri set an alarm at 7am", the phone briefly shows the Siri vortex and then stops. And I find out at 7am when my sad and confused daughter is startled out of bed that her stupid HomePod upstairs on the opposite side the house picked up what must've be…
If Face ID gets a glance lock — I.e. looking at your iPhone directly — while speaking a command it “locks” onto that device (try that out by saying “hey siri” while using an iPhone). You can also just hold the side key instead of Hey Siri-ing it to not hop onto another mic.