Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a fully local LLM voice assistant to control my smart home

johnthenerd.com

Post reply on HN
Print view View original

31–40 of 194 posts

Re: Building a fully local LLM voice assistant to control my smart home

#31

Post by JohnTheNerd »

wslh wrote: post #12
JohnTheNerd wrote: post #9

Earlier quoted context omitted.

that is correct! I would much rather run everything in-house, where I know the quality won't be degraded over time (see the Google Assistant announcement from yesterday) and I am in full control of my data. using a cloud service is much easier and cheaper, but I was not comfortable with that trade-off.

Based on your experience and existing code, it is easy to add continuous listening? Have not tested it but probably is already there. For example, I would like to have it always turned on and speaking to it about ideas at random times.

I never tried it, but I think it would go very poorly without a wake word of sorts.

HomeAssistant seems to natively support wake words, but I haven't looked into it yet. I simply use my smartwatch (Wear OS supports replacing Google Assistant with HomeAssistant's Assist functionality) to interact with the LLM

Re: Building a fully local LLM voice assistant to control my smart home

#32

Post by canada_dry »

jasonjmcghee wrote: post #28
canada_dry wrote: post #13

I've been testing various LLMs (that can run locally - sans cloud) and (for example) the llava-v1.5-7b-q4 does a decent job for home automation. Example: I give the LLM a range of 'verbal' instructions related to home automation to see how well they can identify the action, timing, and subject: User: in the sentence "in 15 minutes turn off the living room light" output the subject, action, time, and location as json…

Out of curiosity what are you using the vision aspect for? Fwiw bakllava is a much more recent model, using mistral instead of llama. Same size and capabilities

> vision aspect

It checks a webcam feed to tell me the current weather outside (e.g. sunny, snowing) though the language parsing is a more important feature.

> more recent model

Yes... models are coming out quicker every week - it's hard to keep up! But I put this one in place a few months ago and its been working fine for my purposes (basic voice controller home automation).

Re: Building a fully local LLM voice assistant to control my smart home

#33

Post by jorvi »

polishdude20 wrote: post #30
canada_dry wrote: post #13

I've been testing various LLMs (that can run locally - sans cloud) and (for example) the llava-v1.5-7b-q4 does a decent job for home automation. Example: I give the LLM a range of 'verbal' instructions related to home automation to see how well they can identify the action, timing, and subject: User: in the sentence "in 15 minutes turn off the living room light" output the subject, action, time, and location as json…

What about like, if I said "switch off the lamp at 3:45" How would you translate the Json you'd get out of that to get the same output? The subject would be "lamp" . Your app code would need to know that lamp is also light.

LLM just are waayyy too dangerous for something like home automation, until it becomes a lot more certain you can guarantee an output for an input.

A very dumb innocuous example would be you ordering a single pizza for the two of you, then telling the assistant “actually we’ll treat ourselves, make that two”. Assistant corrects the order to two. Then the next time you order a pizza “because I had a bad day at work”, assistant just assumes you ‘deserve’ two even if your verbal command is to order one.

A much scarier example is asking the assistant to “preheat the oven when I move downstairs” a few times. Then finally one day you go on vacation and tell the assistant “I’m moving downstairs” to let it know it can turn everything off upstairs. You pick up your luggage in the hallway none the wiser, leave and.. yeah. Bye oven or bye home.

Edit: enjoy your unlocked doors, burned down homes, emptied powerwalls, rained in rooms! :)

Re: Building a fully local LLM voice assistant to control my smart home

#35

Post by coder543 »

jorvi wrote: post #33
polishdude20 wrote: post #30

Earlier quoted context omitted.

What about like, if I said "switch off the lamp at 3:45" How would you translate the Json you'd get out of that to get the same output? The subject would be "lamp" . Your app code would need to know that lamp is also light.

LLM just are waayyy too dangerous for something like home automation, until it becomes a lot more certain you can guarantee an output for an input. A very dumb innocuous example would be you ordering a single pizza for the two of you, then telling the assistant “actually we’ll treat ourselves, make that two”. Assistant corrects the order to two. Then the next time you order a pizza “because I had a bad day at work”,…

No. LLMs do not have memory like that (yet).

Your 'scary' examples are very hypothetical and would require intentional design to achieve today; they would not happen by accident.

Re: Building a fully local LLM voice assistant to control my smart home

#36

Post by canada_dry »

polishdude20 wrote: post #30
canada_dry wrote: post #13

I've been testing various LLMs (that can run locally - sans cloud) and (for example) the llava-v1.5-7b-q4 does a decent job for home automation. Example: I give the LLM a range of 'verbal' instructions related to home automation to see how well they can identify the action, timing, and subject: User: in the sentence "in 15 minutes turn off the living room light" output the subject, action, time, and location as json…

What about like, if I said "switch off the lamp at 3:45" How would you translate the Json you'd get out of that to get the same output? The subject would be "lamp" . Your app code would need to know that lamp is also light.

User: in the sentence "switch off the lamp at 3:45" output the subject, action, time, and location as json

Llama: { "subject": "lamp", "action": "switch off", "time": "3:45", "location": "" }

Where there is an empty parameter the code will try to look back to the last recent commands for context (e.g. I may have just said "turn on the living room light"). If there's an issue it just asks for the missing info.

Translating the parameters from the json is done with good old fashion brute force (i.e. mostly regex).

It's still not 100% perfect but its faster and more accurate than the cloud assistants and private.

Re: Building a fully local LLM voice assistant to control my smart home

#38

Post by JohnTheNerd »

faeriechangling wrote: post #26
JohnTheNerd wrote: post #17

Earlier quoted context omitted.

yes, they are the 16GB models. beware that the memory bus limits you quite a bit. however, buying brand new, they are the best VRAM per dollar in the NVIDIA world as far as I could see. I use 4-bit GPTQ quants. I use tensor parallelism (vLLM supports it natively) to split the model across two GPUs, leaving me with exactly zero free VRAM. there are many reasons behind this decision (some of which are explained in the…

> however, buying brand new, they are the best VRAM per dollar in the NVIDIA world as far as I could see. 3060 12gb is cheaper upfront and a viable alternative. 3090ti used is also cheaper $/vram although a power hog. 4060 16gb is a nice product, just not for gaming. I would wait for price drops because Nvidia just released the 4070 super which should drive down the cost of the 4060 16gb. I also think the 4070ti supe…

that is true, but consider two things:

- motherboards and CPUs have a limited number of PCIe lanes available. I went with a second-hand Threadripper 2920x to be able to have 4 GPU's in the future. since you can only fit so many GPUs, your total available VRAM and future upgrade capacity is overall limited. these decisions limit me to PCIe gen 3x8 (motherboard only supports PCIe gen 3, and 4060Ti only supports 8 lanes), but I found that it's still quite workable. during regular inference, mixtral 8x7b at 4-bit GPTQ quant using vLLM can output text faster than I can read (maybe that says something about my reading speed rather than the inference speed, though). I average ~17 tokens/second.

- power consumption is big when you are self-hosting. not only when you get the power bill, but also for safety reasons. you need to make sure you don't trip the breaker (or worse!) during inference. the 4060Ti draws 180W at max load. 3090's are also notorious for (briefly) drawing well over their rated wattage, which scared me away.

Re: Building a fully local LLM voice assistant to control my smart home

#39

Post by jorvi »

coder543 wrote: post #35
jorvi wrote: post #33

Earlier quoted context omitted.

LLM just are waayyy too dangerous for something like home automation, until it becomes a lot more certain you can guarantee an output for an input. A very dumb innocuous example would be you ordering a single pizza for the two of you, then telling the assistant “actually we’ll treat ourselves, make that two”. Assistant corrects the order to two. Then the next time you order a pizza “because I had a bad day at work”,…

No. LLMs do not have memory like that (yet). Your 'scary' examples are very hypothetical and would require intentional design to achieve today; they would not happen by accident.

I love how burning your house down is something that deserves air quotes according to you.

All I can tell you is this: LLM’s frequently misinterpret, hallucinate and “lie”.

Good luck.

Re: Building a fully local LLM voice assistant to control my smart home

#40

Post by fragmede »

polishdude20 wrote: post #30
canada_dry wrote: post #13

I've been testing various LLMs (that can run locally - sans cloud) and (for example) the llava-v1.5-7b-q4 does a decent job for home automation. Example: I give the LLM a range of 'verbal' instructions related to home automation to see how well they can identify the action, timing, and subject: User: in the sentence "in 15 minutes turn off the living room light" output the subject, action, time, and location as json…

What about like, if I said "switch off the lamp at 3:45" How would you translate the Json you'd get out of that to get the same output? The subject would be "lamp" . Your app code would need to know that lamp is also light.

In all seriousness, I have names for my lights for this very reason.
Post reply on HN