Live data from Hacker News

Building a fully local LLM voice assistant to control my smart home

johnthenerd.com

151–160 of 194 posts

Re: Building a fully local LLM voice assistant to control my smart home

#151
post #130

Earlier quoted context omitted.

I can write a control system easy enough to do this. I'm kind of an expert at that, for oddball reasons, and that's a trivial amount of work for me. The "smart enough" part, I'm more than smart enough for. What's not a trivial amount of work is figuring out how to integrate that into HA. I can guarantee that there is an uncountably infinite number of people like me, and very few people like you. You don't need to do…

1) You can basically do this today with Blueprints. There's also things like Pyscript [0]. 2) The Node-RED implementation in HA is phenomenal and kids can very easily use with a short introduction. 3) Again, already there. ESPHome is a first class citizen in HA. I feel like you've not read the HA docs [1,] or took the time to understand the architecture [2]. And, for someone who has more than enough self-proclaimed s…

I think we are talking across each other.

(1) You are correct that I have not read the docs or discovered everything there is. I have had HA for a few weeks now. I am figuring stuff out. I am finding the learning curve to be steep.

(2) However, I don't think you understand the level of usability and integration I'm suggesting. For most users, "read the docs" or "there's a github repo somewhere" is no longer a sufficient answer. That worked fine for 1996-era Linux. In 2023, this needs to be integrated into the user interface, and you need discoverability and on-ramps. This means actually treating developers as customers. Take a walk through Micro:bit and MakeCode to understand what a smooth on-ramp looks like. Or the Scratch ecosystem.

This contrasts with the macho "for someone who has more than enough self-proclaimed skills, this should be a very understandable system" -- no, it is not a very understandable system for me. Say what you will about my skills, that means it will also not be an understandable system for most e.g. kids and families.

That said, if you're correct, a lot of this may just be a question of relatively surface user-interface stuff, configuration and providing good in-line documentation.

(3) Skills are not universal. A martial artist might be a great athlete, but unless you're Kareem Abdul-Jabbar, that doesn't make you a great basketball player. My skills do include (1) designing educational experiences for kids; and (2) many semesters of graduate-level coursework on control theory.

That's very different from being fluid at e.g. managing docker containers, which I know next to nothing about. My experience trying to add things to HA has not been positive. I spent a lot of time trying to add extensions which would show me a Zigbee connectivity map to debug some connectivity issues. None worked. I eventually found a page which told me this was already in the system *shrug*. I still don't know why the ones I installed didn't work, or where to get started debugging.

For me, that was harder than doing a root-locus plot, implementing a system identification, designing a lag or lead compensator, or running the Bode obstacle course.

Seriously. If I went into HA, and there was a Python console with clear documentation and examples, this would be built. That's my particular skills, but a userbase brings very diverse other skills.

Re: Building a fully local LLM voice assistant to control my smart home

#153

Earlier quoted context omitted.

Reading this gave me an idea to extend this even further. What if the AI could look at your logbook history and suggest automations? For example, I have an automation that turns the lights on when it's dark based on a light sensor. It would be neat if AI could see "hey, you tend to manually turn on the lights when the light level is below some value, want to create an automation for that?"

Machine learning can tackle this for sure, but that's surely separate to LLMs. A language model deals with language, not logic.

This is a very insightful viewpoint. In this situation, I believe it is necessary to use NER to connect the LLM module and the ML module.

Re: Building a fully local LLM voice assistant to control my smart home

#154
post #52

Founder of Home Assistant here. Great write up! With Home Assistant we plan to integrate similar functionality this year out of the box. OP touches upon some good points that we have also ran into and I would love the local LLM community to solve: * I would love to see a standardized API for local LLMs that is not just a 1:1 copying the ChatGPT API. For example, as Home Assistant talks to a random model, we should be…

I don't suppose you guys have something in the works for a polished voice I/O device to replace Alexa and Google Home? They work fine, but need internet connections to function. If the desire is to move to fully offline capabilities then we need the interface hardware to support. You've already proven you can move in the hardware market (I'm using one of your yellow devices now). I know I'd gladly pay for a fully offline interface for every room of my house.

Re: Building a fully local LLM voice assistant to control my smart home

#155

Earlier quoted context omitted.

Reading this gave me an idea to extend this even further. What if the AI could look at your logbook history and suggest automations? For example, I have an automation that turns the lights on when it's dark based on a light sensor. It would be neat if AI could see "hey, you tend to manually turn on the lights when the light level is below some value, want to create an automation for that?"

I've been working on something like this but it's of course harder than it sounds, mostly due to how few example use cases there are. A dumb false positive for yours might be "you tend to turn off the lights when the outside temperature is 50º" Anyone know of a database of generic automations to train on?

Temperature and light may create illusions in LLM. A potential available solution to this is to establish a knowledge graph based on sensor signals, where LLM is used to understand the speech signals given by humans and then interpret these signals as operations on the graph using similarity calculations.

Re: Building a fully local LLM voice assistant to control my smart home

#156

Was I the only who got to the end and was like, “and then…?” You installed it and customised your prompts and then… it worked? It didn’t work? You added the hugging face voice model? I appreciate the prompt, but broadly speaking it feels like there’s a fair bit of vague hand waving here: did it actually work? It mixtral good enough to consistently respond in an intelligent manner? My experience with this stuff has be…

mixtral 7*8B does indeed have this characteristic. It tends to disregard the requirement for structured output and often outputs unnecessary things in a very casual manner. However, I have found that models like qwen 72b or others have better controllability in this aspect, at least reaching the level of gpt 3.5.

Re: Building a fully local LLM voice assistant to control my smart home

#157
post #121
post #63

Earlier quoted context omitted.

When you think about the damage that could be done with this kind of technology it’s incredible. Imagine asking your MixAIr to sort out some fresh dough in a bole and then leaving your house for a while. It might begin to spin uncontrollably fast and create an awful lot of hyperbole-y activity.

I suggest looking up how electric motors work lest you continue looking stupid :)

[deleted]

Re: Building a fully local LLM voice assistant to control my smart home

#158
post #104

Earlier quoted context omitted.

The 8s latency would be absolutely intolerable to me. Queen experimenting, even getting the speech recognition latency low enough not to be a nuisance is already a problem. I'd be inclined to put a bunch of simple grammar based rules in front of the LLM to handle simple/obvious cases without passing them to the LLM at all to at least reduce the number of cases where the latency is high...

Maybe it could be improved by not including all the details in the original prompt, but dynamically generating them. For example, >user: turn my living room lights off >llm: {action: "lights.turn_off", entity: "living room"} Search available actions and entities using the parameters > user: available actions: [...], available entities: [...]. Which action and target? > llm: {service: "light.turn_off", entity: "light.…

Perhaps. Certainly worth trying, but a query like that is also ripe for short-circuiting with templates. For more complex queries it might well be very helpful, though - every little bit helps.

Another thing worth considering in that respect is that ChatGPT at least understands grammars perfectly well. You can give it a BNF grammar and ask it to follow it, and while it won't do so perfectly, tools like LangChain (or you can roll this yourself), lets you force the LLM to follow the grammar precisely. Combine the two and you can give it requests like "translate the following sentence into this grammar: ...".

I'd also simply cache every input/output pairs, at least outside of longer conversations, as I suspect people will get into the habit of saying certain things, and using certain words - e.g. even with the constraint of Alexa, there are many things I use a much more constrained set of phrases than it can handle for, sometimes just out of habit, sometimes because the voice recognition is more likely to correctly pick up certain words. E.g. I say "turn off downstairs" to turn off everything downstairs before going to bed, and I'm not likely to vary that much. A guest might, but a very large proportion of my requests for Alexa uses maybe 10% of even its constrained vocabulary - a delay is much more tolerable if it's for a steadily diminishing set of outliers as you cache more and more...

(A log like that would also potentially be great to see if you could maybe either produce new rules - even have the LLM try to produce rules - or to fine-tune a smaller/faster model as a 'first pass' - you might even be able to start both in parallel and return early if the first one returns something coherent, assuming you can manage to train it to go "don't know" for queries that are too complex)

Re: Building a fully local LLM voice assistant to control my smart home

#159
post #33

Earlier quoted context omitted.

What about like, if I said "switch off the lamp at 3:45" How would you translate the Json you'd get out of that to get the same output? The subject would be "lamp" . Your app code would need to know that lamp is also light.

LLM just are waayyy too dangerous for something like home automation, until it becomes a lot more certain you can guarantee an output for an input. A very dumb innocuous example would be you ordering a single pizza for the two of you, then telling the assistant “actually we’ll treat ourselves, make that two”. Assistant corrects the order to two. Then the next time you order a pizza “because I had a bad day at work”,…

If it wasn't dangerous enough by default, he specifically instructs it to act as much like a homocidal AI from fiction as possible, and then hooks it up to control his house.

I think there's definitely room for this sort of thing to go badly wrong.

Re: Building a fully local LLM voice assistant to control my smart home

#160
post #52

Founder of Home Assistant here. Great write up! With Home Assistant we plan to integrate similar functionality this year out of the box. OP touches upon some good points that we have also ran into and I would love the local LLM community to solve: * I would love to see a standardized API for local LLMs that is not just a 1:1 copying the ChatGPT API. For example, as Home Assistant talks to a random model, we should be…

Note that if going the constrained grammar route, at least ChatGPT (haven't tested on smaller models) understands BNF variants very well, and you can very much give it a compact BNF-like grammar and ask it to "translate X into grammar Y" and it works quite well even zero-shot. It will not be perfect on its own, but perhaps worth testing whether it's worth actually giving it the grammar you will be constraining its response to.

Depending on how much code/json a given model has been trained on, it may or may not also be worth testing if json is the easiest output format to get decent results for or whether something that reads more like a sentence but is still constrained enough to easily parse into JSON works better.

Post reply on HN