Earlier quoted context omitted.
Update on this: When I enable tensor parallelism in llama.cpp, I see 25-33 t/s. With reasoning effort set to medium, Qwen 3.8 finished the same task that previously took 11 hours in a little over three hours, which is still more than three times what most of the large models required including Opus 4.8, and nine times what GPT 5.5 (the fastest of the models I've used) needed for a similar task. So, it's still not fas…
Thanks for the update! I wonder if a Blackwell GPU would be noticeably faster. Which vendor did you end up using? I want to get a gigabyte one but thats be OOS for months.
This model is far from usable on current AMD or Nvidia 128GB AI machines, IMHO, they just don't have the memory bandwidth, especially since it chews so many tokens for any task. If you want to run this specific model, two (or more) 32GB GPUs with decent memory bandwidth is the right way to do it. It doesn't benefit from the larger memory of the Strix Halo. There's enough room for full context and 8-bit quantized model in 64GB. But, it's really a terrible time to buy hardware. MoE models are a much better fir for the Spark and Strix Halo; you can run Laguna S2.1 (slowly) or one of the Qwen 3.6 MoE fine-tunes (pretty quick). Ling 3.0 Flash also looks promising. Nemotron 3.5 Lightning in the MXFP4 quantization absolutely flies on the Strix Halo at 65-80 t/s, but it's dumb. But, all of those are weaker than Qwen 3.8 27B for coding.