Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a fully local LLM voice assistant to control my smart home

johnthenerd.com

Post reply on HN
Print view View original

101–110 of 194 posts

Re: Building a fully local LLM voice assistant to control my smart home

#101

Post by visarga »

balloob wrote: post #69
nox101 wrote: post #66

Earlier quoted context omitted.

I can't help but think of someone downloading "Best Assistant Ever LLM" which pretends to be good but unlocks the doors for thieves or whatever. Is that a dumb fear? With an app I need to trust the app maker. With an app that takes random LLMs I also need to trust the LLM maker. For text gen, or image gen I don't care but for home automation, suddenly it matters if the LLM unlocks my doors, turns on/off my cameras, t…

That could be solved by using something like Anthropic's Constitutional AI[1]. This works by adding a 2nd LLM that makes sure the first LLM acts according to a set of rules (the constitution). This could include a rule to block unlocking the door unless a valid code has been presented. [1]: https://www-files.anthropic.com/production/images/Anthropic_...

This "second llm" is only used during finetuning, not in deployment.

Re: Building a fully local LLM voice assistant to control my smart home

#102

Post by MrQuincle »

balloob wrote: post #64
balloob wrote: post #52

Founder of Home Assistant here. Great write up! With Home Assistant we plan to integrate similar functionality this year out of the box. OP touches upon some good points that we have also ran into and I would love the local LLM community to solve: * I would love to see a standardized API for local LLMs that is not just a 1:1 copying the ChatGPT API. For example, as Home Assistant talks to a random model, we should be…

I think that LLMs are going to be really great for home automation and with Home Assistant we couldn't be better prepared as a platform for experimentation for this: all your data is local, fully accessible and Home Assistant is open source and can easily be extended with custom code or interface with custom models. All other major smart home platforms limit you in how you can access your own data. Here are some thin…

Retrospective questions would also be really great. Why did the lights not turn off downstairs this night? Or other questions involving history.

Re: Building a fully local LLM voice assistant to control my smart home

#103

Post by abdullin »

Great write-up! It is a pleasure to see more people explore this area.

You can make it even more lean and frugal, if you want.

Here is how we built a voice assistant box for Bashkir language. It is currently deployed at ~10 kindergartens/schools:

1. Run speech recognition and speech generation on server CPU. You need just 3 cores (AMD/Intel) to have fast enough responses. Same for the SBERT embedding models (if your assistant needs to find songs, tales or other resources).

2. Use SaaS LLM for prototyping (e.g. mistral.ai has Mistral small and mistral medium LLMs available via API) or run LLMs on your server via llama.cpp. You'll need more than 3 cores, then.

3. Use ESP32-S3 for the voice box. It is powerful enough to run wake-word model and connect to the server via web sockets.

4. If you want to shape responses in a specific format, review Prompting Guide (especially few-shot prompts) and also apply guidance (e.g. as in Microsoft/Guidance framework). However, normally few-shot samples with good prompts are good enough to produce stable responses on many local LLMs.

NB: We have built that with custom languages that aren't supported by the mainstream models, this involved a bit of fine-tuning and custom training. For the main-steam languages like English, things are way more easy.

This topic fascinates me (also about personal assistants that learn over time). I'm always glad to answer any questions!

Re: Building a fully local LLM voice assistant to control my smart home

#104

Post by vidarh »

JohnTheNerd wrote: post #4
simcop2387 wrote: post #3

I'm working on doing exactly this myself, I'm working on some other stuff related to all this (since I'm also doing other LLM stuff), but nothing published yet. I'm looking at llama.cpp's GBNF grammar support to emulate/simulate some of the function calling needs and I'm planning on using or fine tuning a model like TinyLLama (I don't need the sarcasm abilities of better models) and I'm going to try getting this runn…

I would strongly advise using a GPU for inference. the reason behind this is not mere tokens-per-second performance, but that there is a dramatic difference in how long you have to wait before seeing the first token output . this scales very poorly as your context size increases. since you must feed in your smart home state as part of the prompt, this actually matters quite a bit. another roadblock I ran into is (whi…

The 8s latency would be absolutely intolerable to me. Queen experimenting, even getting the speech recognition latency low enough not to be a nuisance is already a problem.

I'd be inclined to put a bunch of simple grammar based rules in front of the LLM to handle simple/obvious cases without passing them to the LLM at all to at least reduce the number of cases where the latency is high...

Re: Building a fully local LLM voice assistant to control my smart home

#105

Post by Havoc »

geerlingguy wrote: post #74
JohnTheNerd wrote: post #41

Earlier quoted context omitted.

power consumption. I am running multiple GPUs somewhere residential. the 4060Ti only draws 180W at max load (which it almost never reaches). 3090 is about double for 1.5x the VRAM, and it's notorious for briefly consuming much more than its rated wattage. this isn't just about the power bill. consider that your power supply and electrical wiring can only push so many watts. you really don't want to try to draw more t…

A 3090 or 4090 can easily pull down enough power that most consumer UPSes (besides the larger tower ones) will do their 'beep of overload', which at best is annoying, at worst causes stability issues. I think there's a sweet spot around 180-250W for these cards, unless you _really_ need top-end performance.

To me it's the PCI lanes that are the issue. Chances of a random gamer having a PSU that can run dual cards is excellent...chances of dual x16 electrical not so much.

I tried dual in x16 x4 and inference performance cratered versus a single

Re: Building a fully local LLM voice assistant to control my smart home

#106

Post by Havoc »

JohnTheNerd wrote: post #41
Havoc wrote: post #37

Why 4060s? I’d have gone for 2nd hand 3090s personally

power consumption. I am running multiple GPUs somewhere residential. the 4060Ti only draws 180W at max load (which it almost never reaches). 3090 is about double for 1.5x the VRAM, and it's notorious for briefly consuming much more than its rated wattage. this isn't just about the power bill. consider that your power supply and electrical wiring can only push so many watts. you really don't want to try to draw more t…

>3090 is about double for 1.5x the VRAM

Not just that - tensorcore count and memory throughput are both ~triple.

Anyway, don't want to get too hung up on that. Overall looks like a great project & I bet it inspires many here to go down a similar route - congrats.

Re: Building a fully local LLM voice assistant to control my smart home

#107

Post by Havoc »

balloob wrote: post #52

Founder of Home Assistant here. Great write up! With Home Assistant we plan to integrate similar functionality this year out of the box. OP touches upon some good points that we have also ran into and I would love the local LLM community to solve: * I would love to see a standardized API for local LLMs that is not just a 1:1 copying the ChatGPT API. For example, as Home Assistant talks to a random model, we should be…

>Constrained grammars do look like an possible alternative.

I'd suggest combining this with a something like nexusraven. i.e. both constrain it but also have an underlying model fine tuned to output in the required format. That'll improve results and let you use a much smaller model.

Another option is to use two LLMs. One to sus out the users natural lang intent and one to paraphrase the intent into something API friendly. The first model would be more suited to a big generic one, while second would be constrained & HA fine tuned.

Also have a look at project functionary on github - haven't tested it but looks similar.

Re: Building a fully local LLM voice assistant to control my smart home

#109

Post by baobun »

fercircularbuf wrote: post #86

Out of curiosity why the complex networking setup instead of, say, tailscale. What kind of flexibility does it give you that makes up for the infrastructure?

Not OP but I assume it's the security-related "no dependencies on external services or leaking data" requirement.

Even if you'd make an exception for Tailscale, that'd require settonv up and exposing an OIDC provider under a public domain with TLS, which comes with its own complexities.

Re: Building a fully local LLM voice assistant to control my smart home

#110

Post by bambax »

abdullin wrote: post #103

Great write-up! It is a pleasure to see more people explore this area. You can make it even more lean and frugal, if you want. Here is how we built a voice assistant box for Bashkir language. It is currently deployed at ~10 kindergartens/schools: 1. Run speech recognition and speech generation on server CPU. You need just 3 cores (AMD/Intel) to have fast enough responses. Same for the SBERT embedding models (if your…

Is there a more detailed write-up somewhere? I have llama.cpp on a server that I use via a web interface, but what would be the next steps to be able to talk to it? How do you actually connect speech recognition and wake-word on one side, to the server, to speech generation on the other side?
Post reply on HN