121–130 of 139 posts
In case anyone else clicked in wondering what counts as "real time" for this: > On a Pi 5 (16GB), Q3_K_S-2.70bpw [KQ-2] hits 8.03 TPS at 2.70 BPW and maintains 94.18% of BF16 quality. And they talk about other hardware and details. But that's the expanded version of the headline claim.
Earlier quoted context omitted.
The bottleneck in common PC hardware is mostly memory bandwidth. Offloading the computation part to a different chip wouldn’t help if memory access is the bottleneck. There have been a lot of boards and chips for years with dedicated compute hardware, but they’re only so useful for these LLM models that require huge memory bandwidth.
It is also to note that the bandwidth bus has seen very little upgrade over the years and even the onboard RAM on GPU card have seen mediocre upgrades. If everyone and their grandma wasn't using NVidia GPUs we would probably have seen a more competitive market and greater changes outside the chip itself.
We need custom inference chips at scale for this imho. Every computer (whatever formfactor/board) should have an inference unit on it so at least inference is efficient and fast and can be offloaded while the cpu is doing something else.
I can't believe this was downvoted. It makes a lot of sense that it would be highly useful to have mass custom inference chips.
The industry has to copy CUDA, or give up and focus on raster. ASIC solutions are a snipe chase, not to mention small and slow.
Earlier quoted context omitted.
The sota chatbots are getting more and more functionality that is not just LLM inference. They can search the web, process files, integrate with other apps. I think that's why most people will consider local LLMs to be insufficient very soon.
But that's just software that also runs fine locally. A few tools with a local LLM can do it.
Earlier quoted context omitted.
Yes, I thought that too! But qwen3:0.6b (and to some extent gemma 1b) has made me reevaluate. They still aren't useful like large LLMs, but for things like summarization, and other tasks where you can give them structure but want the sheen of natural language they are much better than things like the Phi series were.
That's interesting. For what projects would you want the "sheen of natural language" though?
Earlier quoted context omitted.
"Hey, hey, are you still asleep? Using spare cycles, I have designed an optimal recipe for mashed potatoes, as you mentioned ten days ago. I need you to go get some potatoes."
This sounds a lot like gptars. I want a little gptars tearing around my house. https://youtube.com/shorts/e2t0RxX4b54
Reminds me of a video from the 90's where some wizard put a camcorder and a giant antenna on a petrol powered rc car, an even bigger antenna on his house and controlled it from a 40's style sofa and a huge tube TV in his cramped garage. Over a mile range. Surrounded by enormous cars I think he was going 40-50 mph but with the screaming engine sound and the camera so low to the ground it looked like 500 mph. I'm still laughing, it looked like he was having all of the fun.
Earlier quoted context omitted.
Keen for this also. Been having issues getting a smooth voice experience from HA to ChatGPT. I dont like the whole wakeword concept for the receiver either. I think theres work to be done on the whole stack.
What's wrong with the wakeword stuff? Great timing as I was looking into it yesterday as was thinking about writing my own set of agents to run house stuff. I don't want to spent loads of time on voice interaction so HA wakeword stuff would've been useful. If not I'll bypass HA for voice and really only use HA via mcp. I can do fw dev for micros...but omg do I not want to spend the time looking thru a datasheet and g…
However-> These are device specific. The devices I purchased for this purpose have very few vendor supported wakewords, but even more prominently, refuse to integrate with HA. Possible firmware issue, but I have reloaded the firmware 30 times. I dont necessarily want to purchase something else for this purpose. Which is where building a bespoke HA audio box becomes its own can of worms.
But if you want a custom wake word, or more like a wake phrase, you go down a rabbit hole of training/cost/memory etc that starts to get annoying fast.
I kind of know I am being unreasonable. I dont want a device that just ships off everything it hears to an LLM, even local, that would suck. I just want a third way.
Then theres other stuff. Like HA has a hard time with providing context to an LLM, because it sends the whole conversation thus far off to the LLM for context. It can get really weird really quickly. This caused me a lot of issues with lights for example. It would remember switching a light on, and if that was in the context, would refuse to switch it on a second time if it turned off due to a rule or manual intervention. But if you dont send the context, you cant have deeper conversations. You cant ask subsequent questions basically.
Earlier quoted context omitted.
Doesnt suit my use case sadly.
Back to the drawing board. What about a proximity sensor?
But even thats a bit of a wild tradeoff.
Earlier quoted context omitted.
I've not had any issues with the audio picking up, but its in the living room rather than the kitchen. I have Alexa's in most rooms. I don't play music through it, which I do from the Alexa. Tbh I think the mic and the speakers will be fine when the rest of the 'product' is sorted. I failed to mention I have Claude connected to it rather than their default assistant. To us, this just beats Alexa hands down. I have th…
Interesting, well I'm glad it's working well for you all. I tested with local, HA Cloud, and ChatGPT/Claude and that wasn't the sticking point, it was getting the hardware to hear me or for me to hear it. I will say, while it was too slow (today) with the my local inference hardware (CPU, older computer and a little on my newer MBP) it was magically to talk to and hear back from HA all locally. I look forward to a fu…
This is exactly how I feel. Its also why I like the multiple wake words - one for remote and one for local.
One of the amazing things I've found with the LLM powered voice assistants is being able to 'recover' from mistakes - e.g. when cooking and forgetting to set the next timer, I can recover by asking about another event like when the last timer ended or when I turned off the bedroom light. Its annoying you can't do that with Alexa. This 'complexity' doesn't need a huge or SOTA model to resolve! I also enjoy being able to ask for a song by half title and half description - my wife was trying to play Ghost by Au/Ra, which we just can't get the Alexa to do, and I can't reasonably get my local LLMs to fail at.
After your comment earlier I took the preview edition into the kitchen, where it did perform a lot worse with the multiple bits of white noise and odd room shape.
In case anyone else clicked in wondering what counts as "real time" for this: > On a Pi 5 (16GB), Q3_K_S-2.70bpw [KQ-2] hits 8.03 TPS at 2.70 BPW and maintains 94.18% of BF16 quality. And they talk about other hardware and details. But that's the expanded version of the headline claim.
I tried the q4 quantization when it came out and didn't find it to be great for my coding use case.