Yes, but there's still a difference from what we are worried might be doing with our smartphones, and what we KNOW they are doing with 'voice assistants'.
I turn off voice assistant on my smartphone and don't have any google home/alexa devices either. I have a TV with no voice activation. I won't use voice activation devices.
Is it possible my phone is listening to my anyway and sending recordings to someone? It's possible. If I were smart I would never say anything that would be 'dangerous' if someone heard it with my phone in the room, although Im sure I'm not that careful, it's probably not possible.
If I found out for sure my phone was listening to me surreptitiously, rather than just knowing that it's not impossible -- I'd do something about it.
One affect of arguing "it's all the same, your phone COULD be spying on your voice conversations anyway" is, perhaps ironically, to make people care LESS about it. If it's doing it anyway (or just if I can't prove it's not?), what do I care about Google Voice doing it too? But one disincentive for companies doing it is precisely that people will be upset if they find out. (And they're gonna find out eventually).
You don't need to go live in the woods to care about it. It's good to care about it. You don't need to either "draw the line" at living in the woods without electricity, or draw no lines. We can draw lines. They can move. If your argument is that since you can't be sure that nobody's spying on you, you shouldn't care about the people you DO know are spying on you and shouldn't get upset about it -- that's not how any of this works.
Insisting on all or nothing gets you nothing (cause it turns out 'all' isn't really an option), and who wants nothing?