Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Laguna S 2.1

poolside.ai

Post reply on HN
Print view View original

51–60 of 98 posts

Re: Laguna S 2.1

#54

Post by _mrinalwadhwa_ »

_mrinalwadhwa_ wrote: post #43

Has anyone tried it on a mac yet?

Was able to run it on Apple M3 Max (128 GB)

host: Apple M3 Max, 128 GB model: Laguna-S-2.1, 118B-A8B MoE, Q4_K_M (75 GB), DFlash speculative decoding server: http://127.0.0.1:8000, llama.cpp, ctx 64K, 8-bit KV cache

  mode: max thinking
   #  tokens   tok/s  dflash
   1     600    14.4     11%
   2     600    26.1     27%
   3     600    17.8     18%
   4     600    14.0     16%
   5     600     9.3     15%
  --------------------------------
  median  14.4   mean  16.3   min   9.3   max  26.1   tok/s

  mode: no thinking
   #  tokens   tok/s  dflash
   1     190    10.0     20%
   2     109    26.7     65%
   3      95    29.6     72%
   4      93    32.8     81%
   5     382    14.0     30%
  --------------------------------
  median  26.7   mean  22.6   min  10.0   max  32.8   tok/s

Re: Laguna S 2.1

#57

Post by d2p »

benjiro29 wrote: post #34

!! Be careful when testing the model. A lot of people are testing it, and reporting disappointed results / benchmaxxxing claim. But do not realize that thinking has a issue with the default configuration. Important - make sure that THINKING is enabled. By default it wasn't although I was passing the flag --default-chat-template-kwargs '{"enable_thinking": true}' in vllm recipe. The generation_config.json file that is…

Looks like the default chat template was updated on HF to enable this by default shortly after you posted this :-)

Re: Laguna S 2.1

#59

Post by alfiedotwtf »

verdverm wrote: post #26
mft_ wrote: post #23

Looks impressive, and this size fits achievable home hardware. That said, if someone would kindly quantise this down for the 64GB paupers, that would be appreciated. (I know there’s likely degradation, but some people reported good results with a 2 bit version of Qwen 3.5 122B, and this is starting from a higher point. Would be interesting to try, at least.) Edit: someone in the process of doing so: https://huggingfa…

The tool I've been using, llm-compressor, can quant models that do not fit in memory (use the sequential pipeline) https://github.com/vllm-project/llm-compressor my setup to help you on your way: https://github.com/verdverm/quantr Though it seems these will not be needed as Poolside has published quants & dflash with their models.

Nice! Do you know of any tools that do this for tensorrt models?
Post reply on HN