Earlier quoted context omitted.
I kinda have that though. I use a home assistant voice preview speaker, connected to a locally hosted whisper instance backed by Qwen 3.6. All local and fairly energy-reasonable :) Just saying it's not impossible to have your cake and eat it.
Under a hundred bucks?! Ooooh! https://www.home-assistant.io/voice-pe/
The only issue right now is that home control is relatively slow because home assistant plonks way too much context into the pipeline. It would work much better with an elective tool model like MCP but they don't have this yet.
PS: You could also choose the new Pine64 voice thing, It's looking pretty decent. It wasn't out yet when I bought mine.