Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a fully local LLM voice assistant to control my smart home

johnthenerd.com

Post reply on HN
Print view View original

21–30 of 194 posts

Re: Building a fully local LLM voice assistant to control my smart home

#21

Post by stavros »

AlphaWeaver wrote: post #20
stavros wrote: post #5

I did the same thing, but I went the easy way and used OpenAI's API. Half way through, I got fed up with all the boilerplate, so I wrote a really simple (but very Pythonic) wrapper around function calling with Python functions: https://github.com/skorokithakis/ez-openai Then my assistant is just a bunch of Python functions and a prompt. Very very simple. I used an ESP32-Box with the excellent Willow project for the l…

See Magentic for something similar: https://github.com/jackmpcollins/magentic

That looks very interesting, thanks!

Re: Building a fully local LLM voice assistant to control my smart home

#22

Post by Jedd »

JohnTheNerd wrote: post #17
Jedd wrote: post #14

Really great write-up, thank you John. Two naive questions. First, with the 4060 Ti, are those the 16gb models? (I'm idly comparing pricing in Australia, as I've started toying with LM-Studio and lack of VRAM is, as you say, awful.) Semi-related, the actual quantisation choice you made wasn't specified. I'm guessing 4 or 5 bit? - at which point my question is around what ones you experimented with, after setting up y…

yes, they are the 16GB models. beware that the memory bus limits you quite a bit. however, buying brand new, they are the best VRAM per dollar in the NVIDIA world as far as I could see. I use 4-bit GPTQ quants. I use tensor parallelism (vLLM supports it natively) to split the model across two GPUs, leaving me with exactly zero free VRAM. there are many reasons behind this decision (some of which are explained in the…

Great, thanks. Economics on IT h/w this side of the pond are often extra-complicated. And as a casual watcher of the space it feels like a lot of discussion and focus has turned towards, the past few months, optimising performance. So I'm happy to wait and see a bit longer.

From TFA I'd gone to look up GPTQ and AWQ, and inevitably found a reddit post [0] from a few weeks ago asking if both were now obsoleted by ELX2. (sigh - too much, too quickly) Sounds like vLLM doesn't support that yet anyway. The tuning it seems to offer is probably offset by the convenience of using TheBloke's ready-rolled GGUF's.

[0] https://www.reddit.com/r/LocalLLaMA/comments/18q5zjt/are_gpt...

Re: Building a fully local LLM voice assistant to control my smart home

#23

Post by cjbprime »

If Mixtral doesn't support system prompts, and you just copy in your system prompts as another "user" message, does that suggest that Mixtral is less resilient to prompt injection than commercial models, because it doesn't have any concept of "trust this instruction more than this other class of instruction"?

Re: Building a fully local LLM voice assistant to control my smart home

#24

Post by gerdesj »

Thank you so much for this write up mate.

I'm fine with the usual systems n networking stuff but the AI bits and bobs is a bit of a blur to me, so having a template to start off with is a bit of a God's send.

I'm a bit of a Home Assistant fan boi. I have eight of them to look after now. They are so useful as a "box that does stuff" on customer sites. I generally deploy HA Supervised to get a full Linux box underneath on a laptop with some USB dongles but the HAOS all in one thing is ideal for a VM.

Anyway, it looks like I have another project at work 8)

Re: Building a fully local LLM voice assistant to control my smart home

#25

Post by simcop2387 »

JohnTheNerd wrote: post #4
simcop2387 wrote: post #3

I'm working on doing exactly this myself, I'm working on some other stuff related to all this (since I'm also doing other LLM stuff), but nothing published yet. I'm looking at llama.cpp's GBNF grammar support to emulate/simulate some of the function calling needs and I'm planning on using or fine tuning a model like TinyLLama (I don't need the sarcasm abilities of better models) and I'm going to try getting this runn…

I would strongly advise using a GPU for inference. the reason behind this is not mere tokens-per-second performance, but that there is a dramatic difference in how long you have to wait before seeing the first token output . this scales very poorly as your context size increases. since you must feed in your smart home state as part of the prompt, this actually matters quite a bit. another roadblock I ran into is (whi…

Ultimately yes I'll be using a GPU. I've got 4x NVIDIA Tesla P40s, 2x A4000 and an A5000 for doing all this. I've already got some things i'm building for the "one client at a time" thing with llama.cpp but it won't really be too important because there's not going to be more than just me using it as a smart home assistant. The SBC comment is around something like an Orange PI 5 which can actually run some stuff on the GPU actually and I want to see if I can get a very low power but "fast enough" system going for it, and use the bigger power hungry GPUs for larger tasks but it's all stuff to play with really.

Re: Building a fully local LLM voice assistant to control my smart home

#26

Post by faeriechangling »

JohnTheNerd wrote: post #17
Jedd wrote: post #14

Really great write-up, thank you John. Two naive questions. First, with the 4060 Ti, are those the 16gb models? (I'm idly comparing pricing in Australia, as I've started toying with LM-Studio and lack of VRAM is, as you say, awful.) Semi-related, the actual quantisation choice you made wasn't specified. I'm guessing 4 or 5 bit? - at which point my question is around what ones you experimented with, after setting up y…

yes, they are the 16GB models. beware that the memory bus limits you quite a bit. however, buying brand new, they are the best VRAM per dollar in the NVIDIA world as far as I could see. I use 4-bit GPTQ quants. I use tensor parallelism (vLLM supports it natively) to split the model across two GPUs, leaving me with exactly zero free VRAM. there are many reasons behind this decision (some of which are explained in the…

> however, buying brand new, they are the best VRAM per dollar in the NVIDIA world as far as I could see.

3060 12gb is cheaper upfront and a viable alternative. 3090ti used is also cheaper $/vram although a power hog.

4060 16gb is a nice product, just not for gaming. I would wait for price drops because Nvidia just released the 4070 super which should drive down the cost of the 4060 16gb. I also think the 4070ti super 16gb is nice for hybrid gaming/llm usage.

Re: Building a fully local LLM voice assistant to control my smart home

#27

Post by alchemist1e9 »

cloudking wrote: post #19

Now someone package this up into a slick software + hardware device please.

I’ve been thinking recently if maybe this is the turning point where open source software can enable mass competition with hardware vendors for a home “brain” that is installed in your mechanical space. For instance what if running self hosted LLMs that will be compute and power hungry is what turns computers for the home into the next appliance. Maybe it’s silly but something about it is giving me this reoccurring vision of a computer appliance in my basement, perhaps in line with my water heater to harness waste heat from the GPUs, and with a patch panel of HDMI/DP ports and maybe audio ports. Instead of looking like today’s computers it looks more like a furnace or box with sleds for GPUs, almost like a blade system.

Re: Building a fully local LLM voice assistant to control my smart home

#28

Post by jasonjmcghee »

canada_dry wrote: post #13

I've been testing various LLMs (that can run locally - sans cloud) and (for example) the llava-v1.5-7b-q4 does a decent job for home automation. Example: I give the LLM a range of 'verbal' instructions related to home automation to see how well they can identify the action, timing, and subject: User: in the sentence "in 15 minutes turn off the living room light" output the subject, action, time, and location as json…

Out of curiosity what are you using the vision aspect for?

Fwiw bakllava is a much more recent model, using mistral instead of llama. Same size and capabilities

Re: Building a fully local LLM voice assistant to control my smart home

#29

Post by Lienetic »

gerdesj wrote: post #24

Thank you so much for this write up mate. I'm fine with the usual systems n networking stuff but the AI bits and bobs is a bit of a blur to me, so having a template to start off with is a bit of a God's send. I'm a bit of a Home Assistant fan boi. I have eight of them to look after now. They are so useful as a "box that does stuff" on customer sites. I generally deploy HA Supervised to get a full Linux box underneath…

Can you share a bit more about why you're deploying HA in customer sites? I'm also a fan of HA and am interested to learn more about what you're doing and how it's going!

Re: Building a fully local LLM voice assistant to control my smart home

#30

Post by polishdude20 »

canada_dry wrote: post #13

I've been testing various LLMs (that can run locally - sans cloud) and (for example) the llava-v1.5-7b-q4 does a decent job for home automation. Example: I give the LLM a range of 'verbal' instructions related to home automation to see how well they can identify the action, timing, and subject: User: in the sentence "in 15 minutes turn off the living room light" output the subject, action, time, and location as json…

What about like, if I said "switch off the lamp at 3:45"

How would you translate the Json you'd get out of that to get the same output? The subject would be "lamp" . Your app code would need to know that lamp is also light.

Post reply on HN