Building a fully local LLM voice assistant to control my smart home

We are using inmp441. They work well with ESP IDF libraries shipped by Expressif.

I'm not aware of any detailed write-ups. Mostly gathered information bit by bit.

On a high level here is how it is working for us:

0. When voice assistant device (ESP32) starts, it establishes web-socket connection to the server. 1. ESP32 chip is constantly running wake-word detection (there is one provided out-of-the-box by ESP-IDF framework (by Expressif) 2. Whenever a wake-word is detected (we trained a custom one, but you can use the ones provided by ESP), chip starts sending audio packets to the backend via web-sockets.

3. Backend collects all audio frames until there is a silence (using voice activity detection in Python). As soon as the instruction is over, tell the device to stop listening and:

4. Pass all collected audio segments to speech detection (using python with custom wav2vec). This gives us the text instruction.

5. Given a text instruction, you could trigger locally llama.cpp (or vLLM, if you have a GPU) or call remote API. It all depends on the system. We have a chain of LLM pipelines and RAG that compose our "business logic" across a bunch of AI skills. What's important - there is a text response in the end.

6. Pass the text response to speech-to-text model on the same machine, stream output back to the edge device.

7. Edge device (ESP32) will speak the words or play MP3 file you have sent the url to.

Does this help?

Look for ONVIF Compatibility, that's an IP Camera inter-operation standard, meaning if a camera or NVR or sensor supports ONVIF then they can be controlled by FOSS. There is also FOSS called ONVIF Device Manager that identifies any ONVIF devices on one's LAN, allows one to operate and configure those devices, and for cameras it tells you their potentially non-standard playback URL.

Honor to meet you!

[Anonymous] founder of a similarly high-profile initiative here.

> Creating a prompt to write JSON is possible but need quite an elaborate prompt and even then the LLM can make errors. We want to make sure that all JSON coming out of the model is directly actionable without having to ask the LLM what they might have meant for a specific value

The LLM cannot make errors. The LLM spits out probabilities for the next tokens. What you do with it is up to you. You can make errors in how you handle this.

Standard usages pick the most likely token, or a random token from the top many choices. You don't need to do that. You can pick ONLY words which are valid JSON, or even ONLY words which are JSON matching your favorite JSON format. This is a library which does this:

https://github.com/outlines-dev/outlines

The one piece of advice I will give: Do NOT neuter the AI like OpenAI did. There is a near-obsession to define "AI safety" as "not hurting my feelings" (as opposed to "not hacking my computer," "not launching nuclear missiles," or "not exterminating humanity."). For technical reasons, that makes them work much worse. For practical reasons, I like AIs with humanity and personality (much as the OP has). If it says something offensive, I won't break.

AI safety, in this context, means validating that it's not:

* setting my thermostat to 300 degrees centigrade

* power-cycling my devices 100 times per second to break them

* waking me in the middle of the night

... and similar.

Also:

* Big win if it fits on a single 16GB card, and especially not just NVidia. The cheapest way to run an LLM is an Intel Arc A770 16GB. The second-cheapest is an NVidia 4060 Ti 16GB

* Azure gives a safer (not safe) way of running cloud-based models for people without that. I'm pretty sure there's a business model running these models safely too.

I can write a control system easy enough to do this. I'm kind of an expert at that, for oddball reasons, and that's a trivial amount of work for me. The "smart enough" part, I'm more than smart enough for.

What's not a trivial amount of work is figuring out how to integrate that into HA.

I can guarantee that there is an uncountably infinite number of people like me, and very few people like you. You don't need to do my work for me; you just need to enable me to do it easily. What's really needed are decent APIs. If I go into Settings->Automation, I get a frustrating trigger/condition/action system.

This should instead be:

1) Allow me to write (maximally declarative) Python / JavaScript, in-line, to script HA. To define "maximally declarative," see React / Redux, and how they trigger code with triggers

2) Allow my kid(s) to do the same with Blockly

3) Ideally, start to extend this to edge computing, where I can push some of the code into devices (e.g. integrating with ESPHome and standard tools like CircuitPython and MakeCode).

This would have the upside of also turning HA into an educational tool for families with kids, much like Logo, Microsoft BASIC, HyperCard, HTML 2.0, and other technologies of yesteryear.

Specifically controlling my lights to give constant light was one of the first things I wanted to do with HA, but the learning curve meant there was never enough time. I'm also a big fan of edge code, since a lot of this could happen much more gradually and discreetly. That's especially true for things with motors, like blinds, where a very slow stepper could make it silent.