Building a fully local LLM voice assistant to control my smart home
1–10 of 194 posts
Re: Building a fully local LLM voice assistant to control my smart home
#2Re: Building a fully local LLM voice assistant to control my smart home
#3I'm working on doing exactly this myself, I'm working on some other stuff related to all this (since I'm also doing other LLM stuff), but nothing published yet. I'm looking at llama.cpp's GBNF grammar support to emulate/simulate some of the function calling needs and I'm planning on using or fine tuning a model like TinyLLama (I don't need the sarcasm abilities of better models) and I'm going to try getting this running on a small SBC for fun for it but I'm not there yet either.
This write up looks like it's someone actually having tackled a good bit of what I'm planning to try too, and I'm hoping to build out a bunch of the support for calling different home assistant services, like adding TODO items and calling scripts and automations and as many things as i can think of.
Re: Building a fully local LLM voice assistant to control my smart home
#4I'm working on doing exactly this myself, I'm working on some other stuff related to all this (since I'm also doing other LLM stuff), but nothing published yet. I'm looking at llama.cpp's GBNF grammar support to emulate/simulate some of the function calling needs and I'm planning on using or fine tuning a model like TinyLLama (I don't need the sarcasm abilities of better models) and I'm going to try getting this runn…
I would strongly advise using a GPU for inference. the reason behind this is not mere tokens-per-second performance, but that there is a dramatic difference in how long you have to wait before seeing the first token output. this scales very poorly as your context size increases. since you must feed in your smart home state as part of the prompt, this actually matters quite a bit.
another roadblock I ran into is (which may not matter to you) that llama.cpp's OpenAI-compatible server only serves one client at a time, while vLLM can do multiple (the KV cache will bleed over to RAM if it won't fit in VRAM, which will destroy performance, but it will at least work). this might be important if you have more than one person using the assistant, because a doubling of response time is likely to make it unusable (I already found it quite slow, at ~8 seconds between speaking my prompt and hearing the first word output).
if you're looking at my fork for the HomeAssistant integration, you probably won't need my authorization code and can simply ignore that commit. I use some undocumented HomeAssistant APIs to provide fine grained access control.
Re: Building a fully local LLM voice assistant to control my smart home
#5I did the same thing, but I went the easy way and used OpenAI's API. Half way through, I got fed up with all the boilerplate, so I wrote a really simple (but very Pythonic) wrapper around function calling with Python functions:
https://github.com/skorokithakis/ez-openai
Then my assistant is just a bunch of Python functions and a prompt. Very very simple.
I used an ESP32-Box with the excellent Willow project for the local speech recognition and generation:
Re: Building a fully local LLM voice assistant to control my smart home
#6I'm working on doing exactly this myself, I'm working on some other stuff related to all this (since I'm also doing other LLM stuff), but nothing published yet. I'm looking at llama.cpp's GBNF grammar support to emulate/simulate some of the function calling needs and I'm planning on using or fine tuning a model like TinyLLama (I don't need the sarcasm abilities of better models) and I'm going to try getting this runn…
I would strongly advise using a GPU for inference. the reason behind this is not mere tokens-per-second performance, but that there is a dramatic difference in how long you have to wait before seeing the first token output . this scales very poorly as your context size increases. since you must feed in your smart home state as part of the prompt, this actually matters quite a bit. another roadblock I ran into is (whi…
you can spawn multiple llama.cpp servers and query them simultaneously. It’s actually better this way because you get to run different models for different purposes or do sanity checks via a second model.
Re: Building a fully local LLM voice assistant to control my smart home
#7I did the same thing, but I went the easy way and used OpenAI's API. Half way through, I got fed up with all the boilerplate, so I wrote a really simple (but very Pythonic) wrapper around function calling with Python functions: https://github.com/skorokithakis/ez-openai Then my assistant is just a bunch of Python functions and a prompt. Very very simple. I used an ESP32-Box with the excellent Willow project for the l…
I assume the issue is about privacy in your case. I am not using Alexa, Siri, etc.
Re: Building a fully local LLM voice assistant to control my smart home
#8Earlier quoted context omitted.
I would strongly advise using a GPU for inference. the reason behind this is not mere tokens-per-second performance, but that there is a dramatic difference in how long you have to wait before seeing the first token output . this scales very poorly as your context size increases. since you must feed in your smart home state as part of the prompt, this actually matters quite a bit. another roadblock I ran into is (whi…
you can spawn multiple llama.cpp servers and query them simultaneously. It’s actually better this way because you get to run different models for different purposes or do sanity checks via a second model.
that is correct, however I am already using all of my VRAM. it would mean I have to degrade my model quality. I instead decided that I would rather have one solid model, and have all my use cases tied to that one model. using RAM instead proved to be problematic for the reasons I mentioned above.
if I had any free VRAM at all, I would fit faster-whisper before I touch any other LLM lol
Re: Building a fully local LLM voice assistant to control my smart home
#9I did the same thing, but I went the easy way and used OpenAI's API. Half way through, I got fed up with all the boilerplate, so I wrote a really simple (but very Pythonic) wrapper around function calling with Python functions: https://github.com/skorokithakis/ez-openai Then my assistant is just a bunch of Python functions and a prompt. Very very simple. I used an ESP32-Box with the excellent Willow project for the l…
I assume the issue is about privacy in your case. I am not using Alexa, Siri, etc.
that is correct! I would much rather run everything in-house, where I know the quality won't be degraded over time (see the Google Assistant announcement from yesterday) and I am in full control of my data.
using a cloud service is much easier and cheaper, but I was not comfortable with that trade-off.
Re: Building a fully local LLM voice assistant to control my smart home
#10I did the same thing, but I went the easy way and used OpenAI's API. Half way through, I got fed up with all the boilerplate, so I wrote a really simple (but very Pythonic) wrapper around function calling with Python functions: https://github.com/skorokithakis/ez-openai Then my assistant is just a bunch of Python functions and a prompt. Very very simple. I used an ESP32-Box with the excellent Willow project for the l…
> > Building a fully local LLM voice assistant
> I did the same thing, but I went the easy way and used OpenAI's API.
This is a cool project, but it's not really the same thing. The #1 requirement that OP had was to not talk to any cloud services ("no exceptions"), and that's the primary reason why I clicked on this thread. I'd love to replace my Google Home, but not if OpenAI just gets to hoover up the data instead.