Until we have a home assistant that can be operated without a connection to the internet without going to the cloud there is always going to be this risk. I don't see any of the big players actually designing a system like this other than Apple.
Apple doesn't really have a good track record with their data sharing without opt-outs in Siri :(