Building a fully local LLM voice assistant to control my smart home

This is oh so very much a strawman. There is rapid progress in AI. For my domains, the first useful model (without finetuning or additional training) was GPT3, which was released in 2020, and had 175B parameters.

We've had three years of optimization on the models, as well as a lot of progress on how to use them. That means we need fewer parameters today than we did in 2020. That doesn't imply there isn't a hard lower bound somewhere. We just don't know where or what it is.

My expectation is we'll continue to do better and better until, where e.g. a 2030 1B parameter model will be competitive with a 2020 200B parameter model, and a 2030 200B parameter model will be much better than either. After some amount of progress, we'll hit it (or more accurately, asymptotically converge to it).

I don't use local LLMs for coding, but for things related to text (it is a large LANGUAGE model, after all). For that, 7B parameter models became adequate sometime in 2023. For reference, in 2020, they were complete nonsense. You'd get cycles of repeating text, or just lose coherence after a sentence or two.

With my setup, local models aren't anywhere close to fast enough for real-time use. For coding, I need real-time use. It wouldn't surprise me if that domain needed more parameters, just based on what I've seen, but I could be proven wrong. If you buy me an H100, I can experiment with it too. As a footnote, many LARGE models work horribly for coding too; OpenAI did a very good job with GPT there (and I haven't used it enough to know, but I've heard Google did too from people who've used Bard).

- read access is mostly available for sensors, even if access wasn't granted.

- some integrations (especially custom integrations) don't care about authorization. my fork mentioned in the blog does, because I explicitly added logic to authorize requests. the HomeAssistant authorization documentation is outdated and no longer works. I looked through the codebase to find extensions that implement it for an example. maybe I should submit a PR that fixes the doc...

- each entity needs to be explicitly allowed. this results in a massive JSON file.

- it needs a custom group added to the .storage/auth file. this is very much not officially supported. however, it has survived every update I have received so far (and I always update HomeAssistant)

I will share what I did in detail when I get some time on my hands

I already had a few entities I didn't really need it using (not for security reasons, but to shorten the system prompt). I simply excluded them within the Jinja template itself. I can see this being a problem with people who have their ovens or thermostats on HA, but I don't necessarily think it's an unsolvable issue if we implement sensible sanity checks on the output.

hilariously, the model I'm using doesn't even have any RLHF. but I am also not very concerned if GlaDOS decides to turn on the coffee machine. maybe I would be slightly more concerned if I had a smart lock, but I think primitive methods such as "throw big rock at window" would be far easier for a bad person.

when it comes to jailbreak prompts, you need to be able to call the assistant in the first place. if you are authorized to call the HomeAssistant API, why would you bother with the LLM? just call the respective API directly and do whatever evil thing you had in mind. I took an unreasonable number of measures to try to stop this from happening, but I admit that's a risk. however, I don't think that's a risk caused by the LLM, but rather the existence of IoT devices.

I'll come back after I get my training dataset finished.

I really want to standardize a 7b model that you prompt with HTML with details and get pure Json responses.