Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

venturebeat.com

Post reply on HN
Print view View original

271–280 of 286 posts

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#271

Post by rahimnathwani »

suprjami wrote: post #270
rahimnathwani wrote: post #249

Earlier quoted context omitted.

For anyone else trying to run this on a Mac with 32GB unified RAM, this is what worked for me: First, make sure enough memory is allocated to the gpu: sudo sysctl -w iogpu.wired_limit_mb=24000 Then run llama.cpp but reduce RAM needs by limiting the context window and turning off vision support. (And turn off reasoning for now as it's not needed for simple queries.) llama-server \ -hf unsloth/Qwen3.5-35B-A3B-GGUF:UD-Q…

Shouldn't you be using MLX because it's optimised for Apple Silicon? Many user benchmarks report up to 30% better memory usage and up to 50% higher token generation speed: https://reddit.com/r/LocalLLaMA/comments/1fz6z79/lm_studio_s... As the post says, LM Studio has an MLX backend which makes it easy to use. If you still want to stick with llama-server and GGUF, look at llama-swap which allows you to run one fronten…

I didn't know about llama-swap until yesterday. Apparently you can set it up such that it gives different 'model' choices which are the same model with different parameters. So, e.g. you can have 'thinking high', 'thinking medium' and 'no reasoning' versions of the same model, but only one copy of the model weights would be loaded into llama server's RAM.

Regarding mlx, I haven't tried it with this model. Does it work with unsloth dynamic quantization? I looked at mlx-community and found this one, but I'm not sure how it was quantized. The weights are about the same size as unsloth's 4-bit XL model: https://huggingface.co/mlx-community/Qwen3.5-35B-A3B-4bit/tr...

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#272

Post by suprjami »

rahimnathwani wrote: post #271
suprjami wrote: post #270

Earlier quoted context omitted.

Shouldn't you be using MLX because it's optimised for Apple Silicon? Many user benchmarks report up to 30% better memory usage and up to 50% higher token generation speed: https://reddit.com/r/LocalLLaMA/comments/1fz6z79/lm_studio_s... As the post says, LM Studio has an MLX backend which makes it easy to use. If you still want to stick with llama-server and GGUF, look at llama-swap which allows you to run one fronten…

I didn't know about llama-swap until yesterday. Apparently you can set it up such that it gives different 'model' choices which are the same model with different parameters. So, e.g. you can have 'thinking high', 'thinking medium' and 'no reasoning' versions of the same model, but only one copy of the model weights would be loaded into llama server's RAM. Regarding mlx, I haven't tried it with this model. Does it wor…

Yes that's right. The config is described by the developer here:

https://www.reddit.com/r/LocalLLaMA/comments/1rhohqk/comment...

And is in the sample config too:

https://github.com/mostlygeek/llama-swap/blob/main/config.ex...

iiuc MLX quants are not GGUFs for llama.cpp. They are a different file format which you use with the MLX inference server. LM Studio abstracts all that away so you can just pick an MLX quant and it does all the hard work for you. I don't have a Mac so I have not looked into this in detail.

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#273

Post by neop1x »

Aurornis wrote: post #102

If you're new to this: All of the open source models are playing benchmark optimization games. Every new open weight model comes with promises of being as good as something SOTA from a few months ago then they always disappoint in actual use. I've been playing with Qwen3-Coder-Next and the Qwen3.5 models since they were each released. They are impressive, but they are not performing at Sonnet 4.5 level in my experien…

Depends on what you expect from the model. For coding/agentic tasks there is SWE Bench https://www.swebench.com/ which gives a better picture. MiniMax, GLM and Kimi K2 seem to be better models for this purpose than Qwen. And it matches my (limited) actual experience.

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#274

Post by regularfry »

Balinares wrote: post #263
regularfry wrote: post #236

Earlier quoted context omitted.

Currently sat waiting for the unsloth fixed quants to drop, but I'm on the edge of my seat for this.

Wait, didn't they drop like two days ago?

The 35b did but not the 27b. Looks like the latter has been updated in the last half hour.

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#276

Post by kir-gadjello »

Aerroon wrote: post #260
kir-gadjello wrote: post #131

Earlier quoted context omitted.

I just use openrouter, it's free for now. But I would pay 30-100$ to use it 24/7.

Have you tried Minimax M2.5? How did it compare?

Much worse - from my experience minimax is not suitable for high autonomy on hard projects. The real distant second in my experience is mimo flash v2 (but I did not try the latest version, might be closer to parity). I would not use minimax for serious work.

StepFun 3.5 Flash is better compared to google's gemini 3 flash which is surprisingly good and pretty costly, and to GLM-5.

I find this outcome ironic given minimax's more aggressive marketing and large-scale distillation accusations from Anthropic specifically accusing minimax but not StepFun.

I can only wonder about the true underlying reasons, but deducing from public information I suspect that minimax simply has weaker, benchmaxx-targeting post-training R&D and leans more on distillation of western frontier models, while StepFun has extensive post-training with lots of hard-won custom R&D and internal large-scale distillation teachers.

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#277

Post by Aerroon »

kir-gadjello wrote: post #276
Aerroon wrote: post #260

Earlier quoted context omitted.

Have you tried Minimax M2.5? How did it compare?

Much worse - from my experience minimax is not suitable for high autonomy on hard projects. The real distant second in my experience is mimo flash v2 (but I did not try the latest version, might be closer to parity). I would not use minimax for serious work. StepFun 3.5 Flash is better compared to google's gemini 3 flash which is surprisingly good and pretty costly, and to GLM-5. I find this outcome ironic given mini…

Interesting. I'm surprised you feel that it's better than GLM 5 - these models are in different weight classes after all.

I tried it out a bunch and it seems good. I can't really tell if it's better or worse than most of these other models in such a short time though.

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#278

Post by [deleted] »

Aerroon wrote: post #277
kir-gadjello wrote: post #276

Earlier quoted context omitted.

Much worse - from my experience minimax is not suitable for high autonomy on hard projects. The real distant second in my experience is mimo flash v2 (but I did not try the latest version, might be closer to parity). I would not use minimax for serious work. StepFun 3.5 Flash is better compared to google's gemini 3 flash which is surprisingly good and pretty costly, and to GLM-5. I find this outcome ironic given mini…

Interesting. I'm surprised you feel that it's better than GLM 5 - these models are in different weight classes after all. I tried it out a bunch and it seems good. I can't really tell if it's better or worse than most of these other models in such a short time though.

[deleted]

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#279

Post by kir-gadjello »

Aerroon wrote: post #277
kir-gadjello wrote: post #276

Earlier quoted context omitted.

Much worse - from my experience minimax is not suitable for high autonomy on hard projects. The real distant second in my experience is mimo flash v2 (but I did not try the latest version, might be closer to parity). I would not use minimax for serious work. StepFun 3.5 Flash is better compared to google's gemini 3 flash which is surprisingly good and pretty costly, and to GLM-5. I find this outcome ironic given mini…

Interesting. I'm surprised you feel that it's better than GLM 5 - these models are in different weight classes after all. I tried it out a bunch and it seems good. I can't really tell if it's better or worse than most of these other models in such a short time though.

I don't think it's strictly better than GLM 5, more like they are peers (but in math competitions StepFun is stronger than most), and in my experience have similar coding/bugfix ceiling where world knowledge is not the deciding factor. But I didn't test GLM 5 for more than 30 hours, and my agentic harness (opencode) might be suboptimal - I'm open to the idea that GLM 5 with the right agentic harness is ready for ultra-long autonomy, but I have yet to see it myself.

Where GLM 5 is strictly worse for me though, compared to StepFun, is long-form content generation (planning, research documents) - but this can be said about geminis too and these are obviously very smart models.

Given the free option I'd explore GLM 5 more, but if I had to pay for it myself ofc I'd choose stepfun every time. Basically I think right now the optimal configuration for maximizing output of correct software features per dollar involves using StepFun or its future class competitor for bulk coding and first stage code review.

Maybe I need to write a blogpost about it after all.

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#280

Post by Balinares »

regularfry wrote: post #274
Balinares wrote: post #263

Earlier quoted context omitted.

Wait, didn't they drop like two days ago?

The 35b did but not the 27b. Looks like the latter has been updated in the last half hour.

Neat! Thanks for correcting me there. I'll go and take a look.
Post reply on HN