Live data from Hacker News

Building a fully local LLM voice assistant to control my smart home

johnthenerd.com

171–180 of 194 posts

Re: Building a fully local LLM voice assistant to control my smart home

#171
post #110

Earlier quoted context omitted.

Is there a more detailed write-up somewhere? I have llama.cpp on a server that I use via a web interface, but what would be the next steps to be able to talk to it? How do you actually connect speech recognition and wake-word on one side, to the server, to speech generation on the other side?

I'm not aware of any detailed write-ups. Mostly gathered information bit by bit. On a high level here is how it is working for us: 0. When voice assistant device (ESP32) starts, it establishes web-socket connection to the server. 1. ESP32 chip is constantly running wake-word detection (there is one provided out-of-the-box by ESP-IDF framework (by Expressif) 2. Whenever a wake-word is detected (we trained a custom one…

> Does this help?

Yes, thank you! Great description. Will try! ;-)

Re: Building a fully local LLM voice assistant to control my smart home

#172
post #85

I did this as well. I also ended up writing a classifier using some python library that seems to outperform home assistant's implementation. Not sure what the issue is there. I just followed the instructions from an LLM and the internet.

Could you share more about the classifier you made?

Okay, it's been awhile, but here's what I have:

1. Define intents, notate keywords for intents that consist of a couple of phrases.

2. Tokenize, handle stopwords, replace synonyms, run a spell checker algorithm (get the best match from a fuzzy comparison).

3. Extract intent, process it, get the best matching entity.

Some of the magic numbers had to be hand-cultivated by a suite of tests I used to derive them, but other than that, it feels pretty straightforward.

I don't know anything about ML or classifiers or intents, I'm just a software engineer that got the rough outline from GPT-4 and executed the task.

I also wrote a machine learning classifier, but I didn't like the results. I ended up going with nltk/fuzzywuzzy because I felt the performance was superior for my dataset. Perhaps this is where HA goes wrong.

Anyways, I use porcupine to listen, VAD to actively listen, and local whisper on a 24 core server to transcribe.

Re: Building a fully local LLM voice assistant to control my smart home

#173
post #52

Founder of Home Assistant here. Great write up! With Home Assistant we plan to integrate similar functionality this year out of the box. OP touches upon some good points that we have also ran into and I would love the local LLM community to solve: * I would love to see a standardized API for local LLMs that is not just a 1:1 copying the ChatGPT API. For example, as Home Assistant talks to a random model, we should be…

I only found out about https://www.rabbit.tech/research today and, to be honest, I still don't fully understand its scope. But reading your lines, I think rabbit's approach could be how a local AI based home automation system could work.

I've gone into a frenzy of home automation this week-end, right after seeing the demo video of this "LAM" from Rabbit, thinking about the potential for software there.

Connected a few home cameras and two lights to an LLM, and made a few purchases.

The worst expensive offender being a tiny camera controlled RC Crawler[1]. The idea would for it to "patrol" my home in my name, with a sassy LLM.

1. https://sniclo.com/products/snt-niva-1-43-enano-off-road-803...

Re: Building a fully local LLM voice assistant to control my smart home

#175
post #132

Was I the only who got to the end and was like, “and then…?” You installed it and customised your prompts and then… it worked? It didn’t work? You added the hugging face voice model? I appreciate the prompt, but broadly speaking it feels like there’s a fair bit of vague hand waving here: did it actually work? It mixtral good enough to consistently respond in an intelligent manner? My experience with this stuff has be…

> no matter how careful your prompt engineering is, you simply can’t use tiny models to do big complicated tasks. I can and do! The progress in ≈7B models has been nothing short of astonishing. > My experience with this stuff has been mixed That's a more accurate way to describe it. I haven't figured out a way to use ≈7B models for many specific tasks. I've followed a rapidly growing number of domains where people ha…

Which domains?

Re: Building a fully local LLM voice assistant to control my smart home

#176
post #151

Earlier quoted context omitted.

1) You can basically do this today with Blueprints. There's also things like Pyscript [0]. 2) The Node-RED implementation in HA is phenomenal and kids can very easily use with a short introduction. 3) Again, already there. ESPHome is a first class citizen in HA. I feel like you've not read the HA docs [1,] or took the time to understand the architecture [2]. And, for someone who has more than enough self-proclaimed s…

I think we are talking across each other. (1) You are correct that I have not read the docs or discovered everything there is. I have had HA for a few weeks now. I am figuring stuff out. I am finding the learning curve to be steep. (2) However, I don't think you understand the level of usability and integration I'm suggesting. For most users, "read the docs" or "there's a github repo somewhere" is no longer a suffici…

I think people might be a bit offended by what sounds like arrogance. But I completely agree with your general concern that nuts and bolts of making somebody else's software work is often frustrating, complicated and inaccessible while math, logic and domain knowledge is "easy" for many people and far more generally known. Even to the point that it's often easier to write your own thing than bother to learn about an existing one.

A way I sometimes evaluate whether to implement some feature in my work is the ratio of the work it does for the user to the work the user has to do for it. Adding a page header in MS Word used to have a very low ratio. A web based LLM is at the other extreme. Installing a bunch of interdependent finicky software just to do simple child-level programming for HA seems like a poor ratio too.

Re: Building a fully local LLM voice assistant to control my smart home

#178
post #98

"I expose HomeAssistant to the internet so I can use it remotely without a VPN," I wonder if this is a common use case? I would not want to expose Home Assistant to the internet because it requires trust in HASS that they keep an eye on vulnerabilities and trust in me that i update HASS regularly. Do many Home assistant users do it? I prefer keeping it behind wireguard.

I do it, but I'm completely insane: - I actually stay on top of all patches, including HomeAssistant itself - I run it behind a WAF and IPS. lots of VLANs around. even if you breach a service, you'll probably trip something up in the horrific maze I created - I use 2-factor authentication, even for the limited accounts - Those limited accounts? I use undocumented HomeAssistant APIs to lock them down to specific entit…

> - Those limited accounts? I use undocumented HomeAssistant APIs to lock them down to specific entities

Mind sharing your process to achieve what sounds like successful implementation of the much-requested ACL/RBAC support?

Re: Building a fully local LLM voice assistant to control my smart home

#179
post #132

Was I the only who got to the end and was like, “and then…?” You installed it and customised your prompts and then… it worked? It didn’t work? You added the hugging face voice model? I appreciate the prompt, but broadly speaking it feels like there’s a fair bit of vague hand waving here: did it actually work? It mixtral good enough to consistently respond in an intelligent manner? My experience with this stuff has be…

> no matter how careful your prompt engineering is, you simply can’t use tiny models to do big complicated tasks. I can and do! The progress in ≈7B models has been nothing short of astonishing. > My experience with this stuff has been mixed That's a more accurate way to describe it. I haven't figured out a way to use ≈7B models for many specific tasks. I've followed a rapidly growing number of domains where people ha…

> I can and do!

I’m openly skeptical.

Most examples I’ve seen of this have been frankly rubbish, which has matched my experience closely.

The larger models, like 70B are capable of generating reasonably good structured outputs and some of the smaller ones like codellama are also quite good.

The 7b models are unreliable.

Some trivial tasks (eg. Chatbot) can be done, but most complex tasks (eg. Generating code) require larger models and multiple iterations.

Still, happy to be shown how wrong I am. Post some examples of good stuff you’ve done on /r/localllama

…but so far, beyond porn, the 7B models haven’t impressed me.

Examples that actually do useful things are almost always either a) claimed with no way of verifying or doing it yourself, or b) actually use the openAI API.

That’s been my experience anyway.

I standby what I said: prompt engineering can only take you so far. There’s a quantitative hard limit on what you can do with just a prompt.

Proof: if it was false, you could do what GPT4 does with 10 param model and a good prompt.

You can’t.

Re: Building a fully local LLM voice assistant to control my smart home

#180
post #151

Earlier quoted context omitted.

I think we are talking across each other. (1) You are correct that I have not read the docs or discovered everything there is. I have had HA for a few weeks now. I am figuring stuff out. I am finding the learning curve to be steep. (2) However, I don't think you understand the level of usability and integration I'm suggesting. For most users, "read the docs" or "there's a github repo somewhere" is no longer a suffici…

I think people might be a bit offended by what sounds like arrogance. But I completely agree with your general concern that nuts and bolts of making somebody else's software work is often frustrating, complicated and inaccessible while math, logic and domain knowledge is "easy" for many people and far more generally known. Even to the point that it's often easier to write your own thing than bother to learn about an…

Thank you so much for that comment. I really appreciate the feedback.

I do sometimes come off as arrogant. That's unfortunate, and in part due to my cultural background. It's helpful feedback. It's difficult to be downvoted or attacked, and not know why.

I will mention: They're just different skill sets. I know people who can dive into a complex piece of code or dev-ops infrastructure and understand it in hours or days. I'm just not one of them.

Learning to design control systems is a very deep (and rather obscure) pile of mathematics which takes many years of study and is a highly specialized. I picked it up for oddball reasons a few decades ago. Doing proper control systems requires a lot of advanced linear algebra, rational functions, frequency domain analysis, parts of differential equations, etc. That's not the same thing as general math skills. Most people who specialize in this field work in Matlab, wouldn't know what docker is, and in terms of general mathematics, have never taken a course on abstract algebra or topology. Even something like differential equations, one needs only a surface understanding of (it disappears when one shifts to Laplace domain or matrix state space representations).

There's a weird dynamic where things we can't do often seem easier or harder than ones we can. Here, I just have a specialized skillset relevant to the conversation. That doesn't imply I'm a genius, or even a brilliant mathematician.

That just implies I can design an optimal control system. Especially for a system with dynamics as simple as room lighting. And would have a fun time doing that for everything in HA in my house and sharing.

I'd really like to have other things work the same way too, for that matter, where e.g. my HVAC runs heating 24/7 at the right level, rather than toggling on and off. With my background, the intimidating part isn't the math or the electronics, but the dev-ops.

Post reply on HN