Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lemonade by AMD: a fast and open source local LLM server using GPU and NPU

lemonade-server.ai

Post reply on HN
Print view View original

81–90 of 133 posts

Re: Lemonade by AMD: a fast and open source local LLM server using GPU and NPU

#81

Post by naasking »

metalliqaz wrote: post #68
hrmtst93837 wrote: post #63

Earlier quoted context omitted.

[flagged]

I was talking about ROCm vs Vulkan. On AMD GPUs, Vulkan has been commonly recognized as the faster API for some time. Both have been slower than CUDA due to most of the hosting projects focusing entirely on Nvidia. Parent post seemed to indicate that newer ROCm releases are better.

Yes, Vulkan is currently faster due to some ROCm regressions: https://github.com/ROCm/ROCm/issues/5805#issuecomment-414161...

ROCm should be faster in the end, if they ever fix those issues.

Re: Lemonade by AMD: a fast and open source local LLM server using GPU and NPU

#85

Post by naasking »

jcgrillo wrote: post #67
cpburns2009 wrote: post #19

Just in case anyone isn't aware. NPUs are low power, slow, and meant for small models.

I wonder what was the imagined use case? TBH I was seriously thinking about buying a framework desktop but the NPU put me off.. I don't get why I should have to pay money for a bunch of silicon that doesn't do anything. And now that there's some software support... it still doesn't do anything? Why does it even exist at all then?

Small models aren't entirely useless, and the NPU can run LLMs up to around 8B parameters from what I've seen. So one way they could be useful: Qwen3 text to speech models are all under 2B parameters, and Open AI's whisper-small speech to text model is under 1B parameters, so you could have an AI agent that you could talk to and could talk back, where, in theory, you could offload all audio-text and text-audio processing to the low power NPU and leave the GPU to do all of the LLM processing.

Re: Lemonade by AMD: a fast and open source local LLM server using GPU and NPU

#86

Post by cpburns2009 »

jcgrillo wrote: post #67
cpburns2009 wrote: post #19

Just in case anyone isn't aware. NPUs are low power, slow, and meant for small models.

I wonder what was the imagined use case? TBH I was seriously thinking about buying a framework desktop but the NPU put me off.. I don't get why I should have to pay money for a bunch of silicon that doesn't do anything. And now that there's some software support... it still doesn't do anything? Why does it even exist at all then?

The NPU is entirely useless for the Framework Desktop, and really all Strix Halo devices. Where it could be useful is cell phones with the examples mentioned by @naasking (audio-text and text-audio processing), and maybe IoT.

Re: Lemonade by AMD: a fast and open source local LLM server using GPU and NPU

#87

Post by zozbot234 »

naasking wrote: post #85
jcgrillo wrote: post #67

Earlier quoted context omitted.

I wonder what was the imagined use case? TBH I was seriously thinking about buying a framework desktop but the NPU put me off.. I don't get why I should have to pay money for a bunch of silicon that doesn't do anything. And now that there's some software support... it still doesn't do anything? Why does it even exist at all then?

Small models aren't entirely useless, and the NPU can run LLMs up to around 8B parameters from what I've seen. So one way they could be useful: Qwen3 text to speech models are all under 2B parameters, and Open AI's whisper-small speech to text model is under 1B parameters, so you could have an AI agent that you could talk to and could talk back, where, in theory, you could offload all audio-text and text-audio proces…

You could always offload some layers to the NPU for lower power use and leave the rest to the GPU. If the latter is power throttled (common for prefill, not for decode) that will be a performance improvement.

Re: Lemonade by AMD: a fast and open source local LLM server using GPU and NPU

#88

Post by UncleOxidant »

dennemark wrote: post #22

I have been using lemonade for nearly a year already. On Strix Halo I am using nothing else - although kyuz0's toolboxes are also nice ( https://kyuz0.github.io/amd-strix-halo-toolboxes/ ) Nowadays you get TTS, STT, text & image generation and image editing should also be possible. Besides being able to run via rocm, vulkan or on CPU, GPU and NPU. Quite a lot of options. They have a quite good and pragmatic pace in d…

How much of a speedup might I get for, say, Qwen3.5-122B if I were to run with lemonade on my Strix Halo vs running it using vulkan with llama.cpp ?

Re: Lemonade by AMD: a fast and open source local LLM server using GPU and NPU

#89

Post by UncleOxidant »

cpburns2009 wrote: post #58
dennemark wrote: post #54

Earlier quoted context omitted.

I have two Strix Halo devices at hand. Privately a framework desktop with 128gb and at work 64GB HP notebook. The 64GB machine can load Qwen3.5 30B-A3B, with VSCode it needs a bit of initial prompt processing to initialize all those tools I guess. But the model is fighting with the other resources that I need. So I am not really using it anymore these days, but I want to experiment on my home machine with it. I just…

Qwen3-Coder-Next works well on my 128GB Framework Desktop. It seems better at coding Python than Qwen3.5 35B-A3B, and it's not too much slower (43 tg/s compared to 55 tg/s at Q4). 27B is supposed to be really good but it's so slow I gave up on it (11-12 tg/s at Q4).

Agreed. Qwen3-coder-next seems like the sweetspot model on my 128GB Framework Desktop. I seem to get better coding results from it vs 27b in addition to it running faster.
Post reply on HN