I mean, it's already happening. People walk around with smartphones, tablets and laptops which constantly record audio for their virtual assistants, and I have not once consented to having my voice recorded by any of those devices.
Has this actually been verified that they are recording without people's knowledge or consent and sending that data home vs just listening locally for a specific wake up word? Don't get me wrong... I think it is horrid and the trend is worrisome, but I haven't seen conclusive proof this is happening yet.
However, it can mishear you sometimes and thereby send a snippet of your conversation to Apple, I suppose.