Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a fully local LLM voice assistant to control my smart home

johnthenerd.com

Post reply on HN
Print view View original

11–20 of 194 posts

Re: Building a fully local LLM voice assistant to control my smart home

#11

Post by stavros »

lolinder wrote: post #10
stavros wrote: post #5

I did the same thing, but I went the easy way and used OpenAI's API. Half way through, I got fed up with all the boilerplate, so I wrote a really simple (but very Pythonic) wrapper around function calling with Python functions: https://github.com/skorokithakis/ez-openai Then my assistant is just a bunch of Python functions and a prompt. Very very simple. I used an ESP32-Box with the excellent Willow project for the l…

> > Building a fully local LLM voice assistant > I did the same thing, but I went the easy way and used OpenAI's API. This is a cool project, but it's not really the same thing. The #1 requirement that OP had was to not talk to any cloud services ("no exceptions"), and that's the primary reason why I clicked on this thread. I'd love to replace my Google Home, but not if OpenAI just gets to hoover up the data instead.

Sure, but the LLM is also the easy part. Mistral is plenty smart for the use case, all you need to do is to use llama.cpp with a JSON grammar and instruct it to return JSON.

Re: Building a fully local LLM voice assistant to control my smart home

#12

Post by wslh »

JohnTheNerd wrote: post #9
wslh wrote: post #7

Earlier quoted context omitted.

I assume the issue is about privacy in your case. I am not using Alexa, Siri, etc.

that is correct! I would much rather run everything in-house, where I know the quality won't be degraded over time (see the Google Assistant announcement from yesterday) and I am in full control of my data. using a cloud service is much easier and cheaper, but I was not comfortable with that trade-off.

Based on your experience and existing code, it is easy to add continuous listening? Have not tested it but probably is already there. For example, I would like to have it always turned on and speaking to it about ideas at random times.

Re: Building a fully local LLM voice assistant to control my smart home

#13

Post by canada_dry »

I've been testing various LLMs (that can run locally - sans cloud) and (for example) the llava-v1.5-7b-q4 does a decent job for home automation.

Example: I give the LLM a range of 'verbal' instructions related to home automation to see how well they can identify the action, timing, and subject:

User: in the sentence "in 15 minutes turn off the living room light" output the subject, action, time, and location as json

Llama: { "subject": "light", "action": "turned off", "time": "15 minutes from now", "location": "living room" }

Several of the latest models are on par to the results from Gpt4 in my tests.

Re: Building a fully local LLM voice assistant to control my smart home

#14

Post by Jedd »

Really great write-up, thank you John.

Two naive questions. First, with the 4060 Ti, are those the 16gb models? (I'm idly comparing pricing in Australia, as I've started toying with LM-Studio and lack of VRAM is, as you say, awful.)

Semi-related, the actual quantisation choice you made wasn't specified. I'm guessing 4 or 5 bit? - at which point my question is around what ones you experimented with, after setting up your prompts / json handling, and whether you found much difference in accuracy between them? (I've been using mistral7b at q5, but running from RAM requires some patience.)

I'd expect a lower quantisation to still be pretty accurate for this use case, with a promise of much faster response times, given you are VRAM-constrained, yeah?

Re: Building a fully local LLM voice assistant to control my smart home

#17

Post by JohnTheNerd »

Jedd wrote: post #14

Really great write-up, thank you John. Two naive questions. First, with the 4060 Ti, are those the 16gb models? (I'm idly comparing pricing in Australia, as I've started toying with LM-Studio and lack of VRAM is, as you say, awful.) Semi-related, the actual quantisation choice you made wasn't specified. I'm guessing 4 or 5 bit? - at which point my question is around what ones you experimented with, after setting up y…

yes, they are the 16GB models. beware that the memory bus limits you quite a bit. however, buying brand new, they are the best VRAM per dollar in the NVIDIA world as far as I could see.

I use 4-bit GPTQ quants. I use tensor parallelism (vLLM supports it natively) to split the model across two GPUs, leaving me with exactly zero free VRAM. there are many reasons behind this decision (some of which are explained in the blog):

- TheBloke's GPTQ quants only support 4-bit and 3-bit. since the quality difference between 3-bit and 4-bit tends to be large, I went with 4-bit. I did not test, but I wanted high accuracy for non-assistant tasks too, so I simply went with 4-bit.

- vLLM only supports GPTQ, AWQ, and SqueezeLM for quantization. vLLM was needed to serve multiple clients at a time and it's very fast (I want to use the same engine for multiple tasks, this smart assistant is only one use case). I get about 17 tokens/second, which isn't great, but very functional for my needs.

- I chose GPTQ over AWQ for reasons I discussed in the post, and don't know anything about SqueezeLM.

Re: Building a fully local LLM voice assistant to control my smart home

#18

Post by Baeocystin »

Jedd wrote: post #14

Really great write-up, thank you John. Two naive questions. First, with the 4060 Ti, are those the 16gb models? (I'm idly comparing pricing in Australia, as I've started toying with LM-Studio and lack of VRAM is, as you say, awful.) Semi-related, the actual quantisation choice you made wasn't specified. I'm guessing 4 or 5 bit? - at which point my question is around what ones you experimented with, after setting up y…

Not specifically related to this project, but I just started playing around with Faraday, and I'm surprised how well my 8GB 3070 does, with even the 20B models. Things are improving rapidly.

Re: Building a fully local LLM voice assistant to control my smart home

#20

Post by AlphaWeaver »

stavros wrote: post #5

I did the same thing, but I went the easy way and used OpenAI's API. Half way through, I got fed up with all the boilerplate, so I wrote a really simple (but very Pythonic) wrapper around function calling with Python functions: https://github.com/skorokithakis/ez-openai Then my assistant is just a bunch of Python functions and a prompt. Very very simple. I used an ESP32-Box with the excellent Willow project for the l…

See Magentic for something similar: https://github.com/jackmpcollins/magentic
Post reply on HN