Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a fully local LLM voice assistant to control my smart home

johnthenerd.com

Post reply on HN
Print view View original

141–150 of 194 posts

Re: Building a fully local LLM voice assistant to control my smart home

#141

Post by seszett »

jorvi wrote: post #120
jodrellblank wrote: post #92

Earlier quoted context omitted.

Chapter 4: In Which Phileas Fogg Astounds Passepartout, His Servant Just as the train was whirling through Sydenham, Passepartout suddenly uttered a cry of despair. "What's the matter?" asked Mr. Fogg. "Alas! In my hurry—I—I forgot—" "What?" "To turn off the gas in my room!" "Very well, young man," returned Mr. Fogg, coolly; "it will burn—at your expense." - Around The World in 80 Days by Jules Verne, who knew that l…

[flagged]

We might have different ovens but I don't see why mine would burn down my house when left on during vacations, but not when baking things for several hours.

Once warm, it doesn't just get hotter and hotter, it keeps the temp I asked for.

Re: Building a fully local LLM voice assistant to control my smart home

#142

Post by darkwater »

balloob wrote: post #52

Founder of Home Assistant here. Great write up! With Home Assistant we plan to integrate similar functionality this year out of the box. OP touches upon some good points that we have also ran into and I would love the local LLM community to solve: * I would love to see a standardized API for local LLMs that is not just a 1:1 copying the ChatGPT API. For example, as Home Assistant talks to a random model, we should be…

That's great news but... Won't make HW requirements for HA way way higher? Thanks for Home Assistant anyway, I'm an avid user!

Re: Building a fully local LLM voice assistant to control my smart home

#143

Post by alright2565 »

darkwater wrote: post #142
balloob wrote: post #52

Founder of Home Assistant here. Great write up! With Home Assistant we plan to integrate similar functionality this year out of the box. OP touches upon some good points that we have also ran into and I would love the local LLM community to solve: * I would love to see a standardized API for local LLMs that is not just a 1:1 copying the ChatGPT API. For example, as Home Assistant talks to a random model, we should be…

That's great news but... Won't make HW requirements for HA way way higher? Thanks for Home Assistant anyway, I'm an avid user!

I've been using HA for years now, and I don't think there's a single feature that's not toggleable. I expect this one to be too, and also hope that LLM offloading to their cloud is part of their paid plan.

Re: Building a fully local LLM voice assistant to control my smart home

#144

Post by alright2565 »

nox101 wrote: post #66
balloob wrote: post #52

Founder of Home Assistant here. Great write up! With Home Assistant we plan to integrate similar functionality this year out of the box. OP touches upon some good points that we have also ran into and I would love the local LLM community to solve: * I would love to see a standardized API for local LLMs that is not just a 1:1 copying the ChatGPT API. For example, as Home Assistant talks to a random model, we should be…

I can't help but think of someone downloading "Best Assistant Ever LLM" which pretends to be good but unlocks the doors for thieves or whatever. Is that a dumb fear? With an app I need to trust the app maker. With an app that takes random LLMs I also need to trust the LLM maker. For text gen, or image gen I don't care but for home automation, suddenly it matters if the LLM unlocks my doors, turns on/off my cameras, t…

HASS breaks things down into "services" (aka actions) and "devices".

If you don't want the LLM to unlock your doors then just don't allow the LLM to call the `lock.unlock` service.

Re: Building a fully local LLM voice assistant to control my smart home

#145

Post by jodrellblank »

jorvi wrote: post #120
jodrellblank wrote: post #92

Earlier quoted context omitted.

Chapter 4: In Which Phileas Fogg Astounds Passepartout, His Servant Just as the train was whirling through Sydenham, Passepartout suddenly uttered a cry of despair. "What's the matter?" asked Mr. Fogg. "Alas! In my hurry—I—I forgot—" "What?" "To turn off the gas in my room!" "Very well, young man," returned Mr. Fogg, coolly; "it will burn—at your expense." - Around The World in 80 Days by Jules Verne, who knew that l…

[flagged]

???

Re: Building a fully local LLM voice assistant to control my smart home

#146

Post by jorvi »

jpsouth wrote: post #140
jorvi wrote: post #121

Earlier quoted context omitted.

I suggest looking up how electric motors work lest you continue looking stupid :)

I’ll just not worry myself over seemingly insane hypotheticals, lest I continue looking stupid, thank you.

I mean there is multiple people all over the main post pointing out how LLMs aren’t reliable but you do you.

Re: Building a fully local LLM voice assistant to control my smart home

#147

Post by weebull »

lhamil64 wrote: post #73
balloob wrote: post #64

Earlier quoted context omitted.

I think that LLMs are going to be really great for home automation and with Home Assistant we couldn't be better prepared as a platform for experimentation for this: all your data is local, fully accessible and Home Assistant is open source and can easily be extended with custom code or interface with custom models. All other major smart home platforms limit you in how you can access your own data. Here are some thin…

Reading this gave me an idea to extend this even further. What if the AI could look at your logbook history and suggest automations? For example, I have an automation that turns the lights on when it's dark based on a light sensor. It would be neat if AI could see "hey, you tend to manually turn on the lights when the light level is below some value, want to create an automation for that?"

Machine learning can tackle this for sure, but that's surely separate to LLMs. A language model deals with language, not logic.

Re: Building a fully local LLM voice assistant to control my smart home

#148

Post by alright2565 »

vidarh wrote: post #104
JohnTheNerd wrote: post #4

Earlier quoted context omitted.

I would strongly advise using a GPU for inference. the reason behind this is not mere tokens-per-second performance, but that there is a dramatic difference in how long you have to wait before seeing the first token output . this scales very poorly as your context size increases. since you must feed in your smart home state as part of the prompt, this actually matters quite a bit. another roadblock I ran into is (whi…

The 8s latency would be absolutely intolerable to me. Queen experimenting, even getting the speech recognition latency low enough not to be a nuisance is already a problem. I'd be inclined to put a bunch of simple grammar based rules in front of the LLM to handle simple/obvious cases without passing them to the LLM at all to at least reduce the number of cases where the latency is high...

Maybe it could be improved by not including all the details in the original prompt, but dynamically generating them. For example,

>user: turn my living room lights off

>llm: {action: "lights.turn_off", entity: "living room"}

Search available actions and entities using the parameters

> user: available actions: [...], available entities: [...]. Which action and target?

> llm: {service: "light.turn_off", entity: "light.living_ceiling"}

I've never used a local LLM, so I don't know what the fixed startup latency is, but this would dramatically reduce the number of tokens required.

Re: Building a fully local LLM voice assistant to control my smart home

#149

Post by shortrounddev2 »

canada_dry wrote: post #13

I've been testing various LLMs (that can run locally - sans cloud) and (for example) the llava-v1.5-7b-q4 does a decent job for home automation. Example: I give the LLM a range of 'verbal' instructions related to home automation to see how well they can identify the action, timing, and subject: User: in the sentence "in 15 minutes turn off the living room light" output the subject, action, time, and location as json…

But why use an llm for that? This kind of intent recognition has existed for a while now and we already have it in the form of smart speakers. It seems like an overkill tool for the job

Re: Building a fully local LLM voice assistant to control my smart home

#150

Post by evmaki »

Awesome write-up - especially the fact that you've gotten it working with good performance locally. It certainly requires a little bit more hardware than your typical home assistant, but I think this will change over time :)

I've been working on this problem in an academic setting for the past year or so [1]. We built a very similar system in a lab at UT Austin and did a user study (demo here https://youtu.be/ZX_sc_EloKU). We brought a bunch of different people in and had them interact with the LLM home assistant without any constraints on their command structure. We wanted to see how these systems might choke in a more general setting when deployed to a broader base of users (beyond the hobbyist/hacker community currently playing with them).

Big takeaways there: we need a way to do long-term user and context personalization. This is both a matter of knowing an individual's preferences better, but also having a system that can reason with better sensitivity to the limitations of different devices. To give an example, the system might turn on a cleaning robot if you say "the dog made a mess in the living room" -- impressive, but in practice this will hurt more than it helps because the robot can't actually clean up that type of mess.

[1] https://arxiv.org/abs/2305.09802

Post reply on HN