For people with AMD card. This is garbage, rocm is garbage. Just install llama.cpp and run llama-server with vulkan option. This is just some slop + JS/Electron garbage put on top.
Lemonade by AMD: a fast and open source local LLM server using GPU and NPU
121–130 of 133 posts
Re: Lemonade by AMD: a fast and open source local LLM server using GPU and NPU
#122[dead]
Re: Lemonade by AMD: a fast and open source local LLM server using GPU and NPU
#123Earlier quoted context omitted.
Small models aren't entirely useless, and the NPU can run LLMs up to around 8B parameters from what I've seen. So one way they could be useful: Qwen3 text to speech models are all under 2B parameters, and Open AI's whisper-small speech to text model is under 1B parameters, so you could have an AI agent that you could talk to and could talk back, where, in theory, you could offload all audio-text and text-audio proces…
That seems like a really niche use case, and probably not worth the surface area? The power savings would have to be truly astonishing to justify it, given what a small fraction of compute time your average device spends processing voice input. I'd wager the 90th percentile siri/ok google/whatever user issues less than 10 voice queries per day. How much power can they use running on normal hardware and how much could…
It's just an example where it fits perfectly, and it's exactly what something like Alexa or Google home needs for low power machine learning, eg. when sitting idle it needs to consume as little power as possible while waiting for a trigger word.
Any context that needs some limited intelligence while consuming little power would benefit from this.
Re: Lemonade by AMD: a fast and open source local LLM server using GPU and NPU
#124Earlier quoted context omitted.
Small models aren't entirely useless, and the NPU can run LLMs up to around 8B parameters from what I've seen. So one way they could be useful: Qwen3 text to speech models are all under 2B parameters, and Open AI's whisper-small speech to text model is under 1B parameters, so you could have an AI agent that you could talk to and could talk back, where, in theory, you could offload all audio-text and text-audio proces…
You could always offload some layers to the NPU for lower power use and leave the rest to the GPU. If the latter is power throttled (common for prefill, not for decode) that will be a performance improvement.
Routing in a MoE model might fit.
Re: Lemonade by AMD: a fast and open source local LLM server using GPU and NPU
#125Re: Lemonade by AMD: a fast and open source local LLM server using GPU and NPU
#126Re: Lemonade by AMD: a fast and open source local LLM server using GPU and NPU
#127I have been using lemonade for nearly a year already. On Strix Halo I am using nothing else - although kyuz0's toolboxes are also nice ( https://kyuz0.github.io/amd-strix-halo-toolboxes/ ) Nowadays you get TTS, STT, text & image generation and image editing should also be possible. Besides being able to run via rocm, vulkan or on CPU, GPU and NPU. Quite a lot of options. They have a quite good and pragmatic pace in d…
[dead]
Re: Lemonade by AMD: a fast and open source local LLM server using GPU and NPU
#128[flagged]
> Been running local LLMs on my 7900 XTX for months and the ROCm experience has been... rough. Just out of curiosity... how so? I only ask because I've been running local models (using Ollama) on my RX 7900 XTX for the last year and a half or so and haven't had a single problem that was ROCm specific that I can think of. Actually, I've barely had any problems at all, other than the card being limited to 24GB of VRAM.…
> I only ask because I've been running local models (using Ollama) on my RX 7900 XTX for the last year and a half or so and haven't had a single problem that was ROCm specific that I can think of.
It's probably using the Vulkan backend, that is pretty stable and performance is good.
Re: Lemonade by AMD: a fast and open source local LLM server using GPU and NPU
#129[flagged]
The only annoyance I've faced waiting for nix to compile a local build. I'd have thought that larger distros had no issues with it.
Re: Lemonade by AMD: a fast and open source local LLM server using GPU and NPU
#130[flagged]