Live data from Hacker News

Lemonade by AMD: a fast and open source local LLM server using GPU and NPU

lemonade-server.ai

131–133 of 133 posts

Re: Lemonade by AMD: a fast and open source local LLM server using GPU and NPU

#131

Earlier quoted context omitted.

You could always offload some layers to the NPU for lower power use and leave the rest to the GPU. If the latter is power throttled (common for prefill, not for decode) that will be a performance improvement.

Routing in a MoE model might fit.

You want routing to be as quick as possible, because there are dependent loads of expert MoE weights (at least from CPU in most setups, potentially from storage) downstream of it. So that ultimately depends on what the bottleneck on that part of the model is: compute, memory throughput or both? If it's throughput, the NPU might be a bad fit.

Re: Lemonade by AMD: a fast and open source local LLM server using GPU and NPU

#132

Earlier quoted context omitted.

> Been running local LLMs on my 7900 XTX for months and the ROCm experience has been... rough. Just out of curiosity... how so? I only ask because I've been running local models (using Ollama) on my RX 7900 XTX for the last year and a half or so and haven't had a single problem that was ROCm specific that I can think of. Actually, I've barely had any problems at all, other than the card being limited to 24GB of VRAM.…

Did you have complete hardware lockups when VRAM is exceeded? I had quite a few on my 7900XTX with llama.cpp (Arch Linux, various driver versions). Once I dial in the quant and context size that never exceed VRAM, it is stable; before that I swear a lot and keep pressing the hardware reset button.

This happens on windows as well for the same reasons so it's not isolated to Rocm and Linux
Post reply on HN