Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a fully local LLM voice assistant to control my smart home

johnthenerd.com

Post reply on HN
Print view View original

131–140 of 194 posts

Re: Building a fully local LLM voice assistant to control my smart home

#131

Post by 05 »

jorvi wrote: post #33
polishdude20 wrote: post #30

Earlier quoted context omitted.

What about like, if I said "switch off the lamp at 3:45" How would you translate the Json you'd get out of that to get the same output? The subject would be "lamp" . Your app code would need to know that lamp is also light.

LLM just are waayyy too dangerous for something like home automation, until it becomes a lot more certain you can guarantee an output for an input. A very dumb innocuous example would be you ordering a single pizza for the two of you, then telling the assistant “actually we’ll treat ourselves, make that two”. Assistant corrects the order to two. Then the next time you order a pizza “because I had a bad day at work”,…

All of those outcomes are already accessible by fat fingering the existing UI. Oven won’t burn your house down, most modern ones will turn off after some preset time, but otherwise you’re just going to overpay for electricity or need to replace the heating element. Unless you have a 5 ton industrial robot connected to your smart home, or have an iron sitting on a pile of clothes plugged in to a smart socket, you’re probably safe.

Re: Building a fully local LLM voice assistant to control my smart home

#132

Post by blagie »

wokwokwok wrote: post #87

Was I the only who got to the end and was like, “and then…?” You installed it and customised your prompts and then… it worked? It didn’t work? You added the hugging face voice model? I appreciate the prompt, but broadly speaking it feels like there’s a fair bit of vague hand waving here: did it actually work? It mixtral good enough to consistently respond in an intelligent manner? My experience with this stuff has be…

> no matter how careful your prompt engineering is, you simply can’t use tiny models to do big complicated tasks.

I can and do! The progress in ≈7B models has been nothing short of astonishing.

> My experience with this stuff has been mixed

That's a more accurate way to describe it. I haven't figured out a way to use ≈7B models for many specific tasks.

I've followed a rapidly growing number of domains where people have figured out how to make them work.

Re: Building a fully local LLM voice assistant to control my smart home

#133

Post by dieantwoord »

balloob wrote: post #52

Founder of Home Assistant here. Great write up! With Home Assistant we plan to integrate similar functionality this year out of the box. OP touches upon some good points that we have also ran into and I would love the local LLM community to solve: * I would love to see a standardized API for local LLMs that is not just a 1:1 copying the ChatGPT API. For example, as Home Assistant talks to a random model, we should be…

I only found out about https://www.rabbit.tech/research today and, to be honest, I still don't fully understand its scope. But reading your lines, I think rabbit's approach could be how a local AI based home automation system could work.

Re: Building a fully local LLM voice assistant to control my smart home

#134

Post by moffkalast »

blagie wrote: post #132
wokwokwok wrote: post #87

Was I the only who got to the end and was like, “and then…?” You installed it and customised your prompts and then… it worked? It didn’t work? You added the hugging face voice model? I appreciate the prompt, but broadly speaking it feels like there’s a fair bit of vague hand waving here: did it actually work? It mixtral good enough to consistently respond in an intelligent manner? My experience with this stuff has be…

> no matter how careful your prompt engineering is, you simply can’t use tiny models to do big complicated tasks. I can and do! The progress in ≈7B models has been nothing short of astonishing. > My experience with this stuff has been mixed That's a more accurate way to describe it. I haven't figured out a way to use ≈7B models for many specific tasks. I've followed a rapidly growing number of domains where people ha…

> The progress in ≈7B models has been nothing short of astonishing.

I'd even still rank Mistral 7B above Mixtral personally, because the inference support for the latter is such a buggy mess that I have yet to get it working consistently and none of what I've seen people claim it can do has ever materialized for me on my local setup. MoE is a real fiddly trainwreck of an architecture. Plus 7B models can run on 8GB LPDDR4X ARM devices at about 2.5 tok/s which might be usable for some integrated applications.

It is rather awesome how far small models have come, though I still remember trying out Vicuna on WASM back in January or February and being impressed enough to be completely pulled into this whole LLM thing. The current 7B are about as good as the 30B were at the time, if not slightly better.

Re: Building a fully local LLM voice assistant to control my smart home

#135

Post by regularfry »

puchatek wrote: post #100
JohnTheNerd wrote: post #62

Earlier quoted context omitted.

thank you for building an amazing product! I suspect cloning OpenAI's API is done for compatibility reasons. most AI-based software already support the GPT-4 API, and OpenAI's official client allows you to override the base URL very easily. a local LLM API is unlikely to be anywhere near as popular, greatly limiting the use cases of such a setup. a great example is what I did, which would be much more difficult witho…

So how much of the 8s is spent in the LLM vs Piper? Some of the example responses are very long for the typical home automation usecase which would compound the problem. Ample room for GladOS to be sassy but at 8s just too tardy to be usable. A different approach might be to use the LLM to produce a set of GladOS-like responses upfront and pick from them instead of always letting the LLM respond with something new. O…

A quick fix for the user experience would be to output a canned "one moment please" as soon as the input's received.

Re: Building a fully local LLM voice assistant to control my smart home

#136

Post by regularfry »

splitrocket wrote: post #81
glenngillen wrote: post #57

Has the state of hobbyist microphone arrays improved? The thing that’s always given me pause here is that my Echo devices are quite good, especially for the cost, at picking things up in a relatively noisy kitchen environment.

100% this. Also, microphones in the wrong room responding. I'm having an issue with that as well.

A few months back I was playing with BLE tokens and espresence receivers so HA can tell which room I'm in. It was way too noisy to be useful at the time, but it strikes me as something that's eminently doable.

Re: Building a fully local LLM voice assistant to control my smart home

#137

Post by [deleted] »

jorvi wrote: post #120
jodrellblank wrote: post #92

Earlier quoted context omitted.

Chapter 4: In Which Phileas Fogg Astounds Passepartout, His Servant Just as the train was whirling through Sydenham, Passepartout suddenly uttered a cry of despair. "What's the matter?" asked Mr. Fogg. "Alas! In my hurry—I—I forgot—" "What?" "To turn off the gas in my room!" "Very well, young man," returned Mr. Fogg, coolly; "it will burn—at your expense." - Around The World in 80 Days by Jules Verne, who knew that l…

[flagged]

[deleted]

Re: Building a fully local LLM voice assistant to control my smart home

#138

Post by herbst »

iamflimflam1 wrote: post #118
herbst wrote: post #115

Earlier quoted context omitted.

Just ordered 2 esp32-s3. Any recommendations for a microphone? I guess that will be the hardest part still

Go for an I2S MEMS microphone. Avoid analog microphones as they'll be very noisy and the ADCs on the ESP32 range are pretty rubbish. You're pretty much limited to PDM microphones nowadays though there are some PCM ones still knocking around. PCM mics are considerably cheaper. Audio is well supported on the ESP32 and there are plenty of libraries and sample code out there.

My last experiments have been with a logitech camera as mic, worked kinda well but unreliable. Seeing forward to the chips ive ordered

Re: Building a fully local LLM voice assistant to control my smart home

#139

Post by windexh8er »

blagie wrote: post #130
balloob wrote: post #76

Earlier quoted context omitted.

That's a good one. We might take it one step further and ask the user if they want to add a rule that certain rooms have a certain level of light. Although light level would tie it to a specific sensor. A smart enough system might also be able to infer this from the position of the sun + weather (ie cloudy) + direction of the windows in the room + curtains open/closed.

I can write a control system easy enough to do this. I'm kind of an expert at that, for oddball reasons, and that's a trivial amount of work for me. The "smart enough" part, I'm more than smart enough for. What's not a trivial amount of work is figuring out how to integrate that into HA. I can guarantee that there is an uncountably infinite number of people like me, and very few people like you. You don't need to do…

1) You can basically do this today with Blueprints. There's also things like Pyscript [0]. 2) The Node-RED implementation in HA is phenomenal and kids can very easily use with a short introduction. 3) Again, already there. ESPHome is a first class citizen in HA.

I feel like you've not read the HA docs [1,] or took the time to understand the architecture [2]. And, for someone who has more than enough self-proclaimed skills, this should be a very understandable system.

[0] https://github.com/custom-components/pyscript [1] https://www.home-assistant.io/docs/ [2] https://developers.home-assistant.io/

Re: Building a fully local LLM voice assistant to control my smart home

#140

Post by jpsouth »

jorvi wrote: post #121
jpsouth wrote: post #63

Earlier quoted context omitted.

When you think about the damage that could be done with this kind of technology it’s incredible. Imagine asking your MixAIr to sort out some fresh dough in a bole and then leaving your house for a while. It might begin to spin uncontrollably fast and create an awful lot of hyperbole-y activity.

I suggest looking up how electric motors work lest you continue looking stupid :)

I’ll just not worry myself over seemingly insane hypotheticals, lest I continue looking stupid, thank you.
Post reply on HN