71–80 of 98 posts
I need to evaluate their large hosted model.
Earlier quoted context omitted.
Seems it’s not fully supported in mainline llama.cpp yet https://github.com/ggml-org/llama.cpp/pull/25165 In the huggingface link they mention building for CPU and for CUDA, does anyone know if that means it wouldn’t be possible to build targeting Vulkan?
Replying to myself, seems this PR was merged into main and it the model does work with a Vulkan backend on my Framework desktop, I’m getting about 220 tok/s prompt processing and 21 tok/s output on the 4-bit quant. This is really a sweet spot imo on this machine between maximizing ram use and still having decent speed due to the expert size. This looks really promising.
Earlier quoted context omitted.
Replying to myself, seems this PR was merged into main and it the model does work with a Vulkan backend on my Framework desktop, I’m getting about 220 tok/s prompt processing and 21 tok/s output on the 4-bit quant. This is really a sweet spot imo on this machine between maximizing ram use and still having decent speed due to the expert size. This looks really promising.
What's your hardware? I've got 64GB RAM and a 4090 here, wondering if it's worth a play.
Unfortunately I don’t know about the best way of running on an Nvidia gpu, you could try llama.cpp and offloading as many layers as possible into the gpu and using RAM for the rest, not sure if that would slow it down too much though.
Looks impressive, and this size fits achievable home hardware. That said, if someone would kindly quantise this down for the 64GB paupers, that would be appreciated. (I know there’s likely degradation, but some people reported good results with a 2 bit version of Qwen 3.5 122B, and this is starting from a higher point. Would be interesting to try, at least.) Edit: someone in the process of doing so: https://huggingfa…
Someone has benchmarked a wide range of different quants.
!! Be careful when testing the model. A lot of people are testing it, and reporting disappointed results / benchmaxxxing claim. But do not realize that thinking has a issue with the default configuration. Important - make sure that THINKING is enabled. By default it wasn't although I was passing the flag --default-chat-template-kwargs '{"enable_thinking": true}' in vllm recipe. The generation_config.json file that is…
Looks like the default chat template was updated on HF to enable this by default shortly after you posted this :-)
Anyone know if this is supported in ik_llama.cpp?
It works, thanks to https://github.com/ikawrakow/ik_llama.cpp/pull/1911 , which got merged in early June. However, there might still be some issue with the chat template.
Edit: it amazes me how fast ik_llama.cpp moves
Earlier quoted context omitted.
> edit: on bigger tests, got it to loop pretty easily unfortunately, probably local settings. Been playing around for a few hours with the poolside/Laguna-S-2.1-NVFP4 + poolside/Laguna-S-2.1-DFlash-NVFP4 + vLLM, been seeing the same behaviour. Usually new model releases are plagued with issues at release though, best to wait 1-2 weeks then retry, or better yet, investigate yourself :) Personally I haven't found any o…
Update: Seems quite literally they have bugs on the hardware I'm trying to run this with: From Poolside CEO Eiso Kant on Twitter: > Learning we have some bugs on the RTX6000. We’re on it. Team has worked non stop last days and it’s getting late for a lot of the inference folks, so might be until tomorrow till we have a solution. - https://x.com/eisokant/status/2079693050796785720 Update2: I'm now running poolside/Lag…
> The BF16 checkpoint doesn't exhibit any of the complaints [...] with our current quants, the models tend to choose the wrong logits sometimes [...] why we'll need a requant [...] We're not aware of any bugs in any runtimes themselves [...] We have two remaining things that we're trying to tackle: some people are reporting thinking being too hard to trigger (^^) , and others are saying it thinks too much. We've seen much more of the latter internally
Earlier quoted context omitted.
> edit: on bigger tests, got it to loop pretty easily unfortunately, probably local settings. Been playing around for a few hours with the poolside/Laguna-S-2.1-NVFP4 + poolside/Laguna-S-2.1-DFlash-NVFP4 + vLLM, been seeing the same behaviour. Usually new model releases are plagued with issues at release though, best to wait 1-2 weeks then retry, or better yet, investigate yourself :) Personally I haven't found any o…
Update: Seems quite literally they have bugs on the hardware I'm trying to run this with: From Poolside CEO Eiso Kant on Twitter: > Learning we have some bugs on the RTX6000. We’re on it. Team has worked non stop last days and it’s getting late for a lot of the inference folks, so might be until tomorrow till we have a solution. - https://x.com/eisokant/status/2079693050796785720 Update2: I'm now running poolside/Lag…
Earlier quoted context omitted.
Update: Seems quite literally they have bugs on the hardware I'm trying to run this with: From Poolside CEO Eiso Kant on Twitter: > Learning we have some bugs on the RTX6000. We’re on it. Team has worked non stop last days and it’s getting late for a lot of the inference folks, so might be until tomorrow till we have a solution. - https://x.com/eisokant/status/2079693050796785720 Update2: I'm now running poolside/Lag…
Update3, summarized from RTX6kPRO Discord: > The BF16 checkpoint doesn't exhibit any of the complaints [...] with our current quants, the models tend to choose the wrong logits sometimes [...] why we'll need a requant [...] We're not aware of any bugs in any runtimes themselves [...] We have two remaining things that we're trying to tackle: some people are reporting thinking being too hard to trigger (^^) , and other…
/* Just started testing with the gguf (with gpu offload, m5 max 128gb), q4_k_m, running seemingly well. Speed is initially slightly faster than antirez/ds4 - decode tok/s in the 30s, prefill ~400-ish. Expected, given the slightly smaller size. Looks to be working fine, but too early to tell. Definitely likes to "think". */
Anyways, guessing that discord might be focusing on the nvfp4 stuff. I've noticed spelling mistakes in the thinking traces, tool calls have been fine so far.