Laguna S 2.1

Post by verdverm » Wed, Jul 22, 2026, 4:56 PM UTC

alfiedotwtf wrote: post #59

verdverm wrote: post #26

Earlier quoted context omitted.

The tool I've been using, llm-compressor, can quant models that do not fit in memory (use the sequential pipeline) https://github.com/vllm-project/llm-compressor my setup to help you on your way: https://github.com/verdverm/quantr Though it seems these will not be needed as Poolside has published quants & dflash with their models.

Nice! Do you know of any tools that do this for tensorrt models?

Post by Lwerewolf » Wed, Jul 22, 2026, 6:54 PM UTC

Lwerewolf wrote: post #80

embedding-shape wrote: post #78

Earlier quoted context omitted.

Update3, summarized from RTX6kPRO Discord: > The BF16 checkpoint doesn't exhibit any of the complaints [...] with our current quants, the models tend to choose the wrong logits sometimes [...] why we'll need a requant [...] We're not aware of any bugs in any runtimes themselves [...] We have two remaining things that we're trying to tackle: some people are reporting thinking being too hard to trigger (^^) , and other…

Deleted earlier, didn't see you post, pasting here: /* Just started testing with the gguf (with gpu offload, m5 max 128gb), q4_k_m, running seemingly well. Speed is initially slightly faster than antirez/ds4 - decode tok/s in the 30s, prefill ~400-ish. Expected, given the slightly smaller size. Looks to be working fine, but too early to tell. Definitely likes to "think". */ Anyways, guessing that discord might be foc…

  Let me also look at the GameLogic::update() to see the rest of the update flow, especially the object update loop.

  Actually, I think I have enough information now. Let me also check the GameLogic::UPDATE to understand the full update flow.

Post by Lwerewolf » Wed, Jul 22, 2026, 9:17 PM UTC

Lwerewolf wrote: post #83

Lwerewolf wrote: post #80

Earlier quoted context omitted.

Deleted earlier, didn't see you post, pasting here: /* Just started testing with the gguf (with gpu offload, m5 max 128gb), q4_k_m, running seemingly well. Speed is initially slightly faster than antirez/ds4 - decode tok/s in the 30s, prefill ~400-ish. Expected, given the slightly smaller size. Looks to be working fine, but too early to tell. Definitely likes to "think". */ Anyways, guessing that discord might be foc…

Well, just ran said gguf on the GeneralsX codebase with a pretty open-ended "Explain this codebase to me, and the general game loop." prompt, and... Let me also look at the GameLogic::update() to see the rest of the update flow, especially the object update loop. Actually, I think I have enough information now. Let me also check the GameLogic::UPDATE to understand the full update flow. ...repeating forever. Since the…

Post by mandelken » Wed, Jul 22, 2026, 9:53 PM UTC

SwellJoe wrote: post #9

This is exactly the kind of model that's been needed in the middle. Realistically self-hosted, Good Enough intelligence, MoE so it's fast on limited bandwidth systems like Strix Halo and DGX Spark. For a while there's been nothing to run on my Strix Halo that's notably better than what I can run on my dual 32GB GPU desktop (Gemma 4 or Qwen 3.6 dense models), but this seems likely to be the step up in size that actual…

Post by Lwerewolf » Thu, Jul 23, 2026, 1:21 PM UTC

embedding-shape wrote: post #86

Lwerewolf wrote: post #85

Earlier quoted context omitted.

Right: https://huggingface.co/poolside/Laguna-S-2.1-FP8/discussions... Wait time.

More updates from the Discord (also mostly about the NVFP4): > we've got an NVFP4 checkpoint that has a KD of 0.135 relative to the BF16 checkpoint (for W4A4, on some GSM8K prompts). [...] We're still seeing some looping on W4A4 (although much less than before), whereas for W4A16 we've not been able to make it loop. Given that W4A4 is still a little broken, we won't make this an official release (we have one more ide…