Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lemonade by AMD: a fast and open source local LLM server using GPU and NPU

lemonade-server.ai

Post reply on HN
Print view View original

21–30 of 133 posts

Re: Lemonade by AMD: a fast and open source local LLM server using GPU and NPU

#21

Post by molticrystal »

JSR_FDED wrote: post #9

I’ve read the website and the news announcement, and I still don’t understand what it is. An alternative to LM Studio? Does it support MLX or metal on Macs? I’m assuming it will optimize things for AMD, but are you at a disadvantage using other GPUs?

>Does it support MLX or metal on Macs?

This is answered from their Project Roadmap over on Github[0]:

Recently Completed: macOS (beta)

Under Development: MLX support

[0] https://github.com/lemonade-sdk/lemonade?tab=readme-ov-file#...

Re: Lemonade by AMD: a fast and open source local LLM server using GPU and NPU

#22

Post by dennemark »

I have been using lemonade for nearly a year already. On Strix Halo I am using nothing else - although kyuz0's toolboxes are also nice (https://kyuz0.github.io/amd-strix-halo-toolboxes/)

Nowadays you get TTS, STT, text & image generation and image editing should also be possible. Besides being able to run via rocm, vulkan or on CPU, GPU and NPU. Quite a lot of options. They have a quite good and pragmatic pace in development. Really recommend this for AMD hardware!

Edit: OpenAI and i think nowaday ollama compatible endpoints allow me to use it in VSCode Copilot as well as i.e. Open Web UI. More options are shown in their docs.

Re: Lemonade by AMD: a fast and open source local LLM server using GPU and NPU

#23

Post by dennemark »

kouunji wrote: post #20

I’m looking forward to trying this currently Strix halo’s npu isn’t accessible if you’re running Linux, and previously I don’t think lemonade was either. If this opens up the npu that would be great! Resolute raccoon is adding npu support as well.

Maybe you have seen NPU support via FLM already: https://lemonade-server.ai/flm_npu_linux.html

"FastFlowLM (FLM) support in Lemonade is in Early Access. FLM is free for non-commercial use, however note that commercial licensing terms apply. "

Re: Lemonade by AMD: a fast and open source local LLM server using GPU and NPU

#24

Post by boomskats »

kouunji wrote: post #20

I’m looking forward to trying this currently Strix halo’s npu isn’t accessible if you’re running Linux, and previously I don’t think lemonade was either. If this opens up the npu that would be great! Resolute raccoon is adding npu support as well.

I thought the NPU has been available since something like 6.12?

Re: Lemonade by AMD: a fast and open source local LLM server using GPU and NPU

#25

Post by rpdillon »

Been running lemonade for some time on my Strix Halo box. It dispatches out to other backends that they include, like diffusion and llama. I actually don't like their combined server, and what I use instead is their llama CPP build for ROCm.

https://github.com/lemonade-sdk/llamacpp-rocm

But I'm not doing anything with images or audio. I get about 50 tokens a second with GPT OSS 120B. As others have pointed out, the NPU is used for low-powered, small models that are "always on", so it's not a huge win for the standard chatbot use case.

Re: Lemonade by AMD: a fast and open source local LLM server using GPU and NPU

#26

Post by zenoprax »

freedomben wrote: post #11
jmillikin wrote: post #7

Surprising that the Linux setup instructions for the server component don't include Docker/Podman as an option, its Snap/PPA for Ubuntu and RPM for Fedora. Maybe the assumption is that container-oriented users can build their own if given native packages?

They do have some container options, though I definitely think they should be added to the release page: https://lemonade-server.ai/install_options.html#docker

Why should this be on the "Releases"? Shouldn't that just be for build artifacts? Pre-built containers belong on a registry, no?

I suppose a Dockerfile could be included but that also seems unconventional.

Re: Lemonade by AMD: a fast and open source local LLM server using GPU and NPU

#27

Post by freedomben »

zenoprax wrote: post #26
freedomben wrote: post #11

Earlier quoted context omitted.

They do have some container options, though I definitely think they should be added to the release page: https://lemonade-server.ai/install_options.html#docker

Why should this be on the "Releases"? Shouldn't that just be for build artifacts? Pre-built containers belong on a registry, no? I suppose a Dockerfile could be included but that also seems unconventional.

I just meant on the instructions part of the releases page (since they already have some installation instructions), not the artifacts themselves.

Re: Lemonade by AMD: a fast and open source local LLM server using GPU and NPU

#29

Post by zozbot234 »

rpdillon wrote: post #25

Been running lemonade for some time on my Strix Halo box. It dispatches out to other backends that they include, like diffusion and llama. I actually don't like their combined server, and what I use instead is their llama CPP build for ROCm. https://github.com/lemonade-sdk/llamacpp-rocm But I'm not doing anything with images or audio. I get about 50 tokens a second with GPT OSS 120B. As others have pointed out, the N…

Even small NPUs can offload some compute from prefill which can be quite expensive with longer contexts. It's less clear whether they can help directly during decode; that depends on whether they can access memory with good throughput and do dequant+compute internally, like GPUs can. Apple Neural Engine only does INT8 or FP16 MADD ops, so that mostly doesn't help.
Post reply on HN