Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers • Hacker News Board

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#241

Post by nu11ptr » Sun, Mar 01, 2026, 2:32 PM UTC

cmenge wrote: post #118

nu11ptr wrote: post #65

Thinking about getting a new MBP M5 Max 128GB (assuming they are released next week). I know "future proofing" at this stage is near impossible, but for writing Rust code locally (likely using Qwen 3.5 for now on MLX), the AIs have convinced me this is probably my best choice for immediate with some level of longevity, while retaining portability (not strictly needed, but nice to have). Alternatively was considering…

I've been mulling the same, but decided against (for now) Using Claude Code Max 20 so ROI would be maybe 2+ years. CC gives me unlimited coding in 4-6 windows in parallel. Unsure if any model would beat (or even match) that, both in terms in quality and speed. I wouldn't gamble on that now. With a subscription, I can change any time. With the machine, you risk that this great insane model comes out but you need 138GB…

We are on the same wavelength. I'm thinking maybe a pass for now.

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#242

Post by nu11ptr » Sun, Mar 01, 2026, 2:33 PM UTC

shell0x wrote: post #169

nu11ptr wrote: post #65

Thinking about getting a new MBP M5 Max 128GB (assuming they are released next week). I know "future proofing" at this stage is near impossible, but for writing Rust code locally (likely using Qwen 3.5 for now on MLX), the AIs have convinced me this is probably my best choice for immediate with some level of longevity, while retaining portability (not strictly needed, but nice to have). Alternatively was considering…

I have a Mac Studio with 128GB and a M4 Max and I'd recommend it. The power usage is also pretty good, but you may not care if you live somewhere where energy is cheap.

Have you used this for Rust coding by chance? I'm curious how it compares to Opus 4.6. I realize it isn't going to think to the same level, but curious how code quality is for a more straight forward task.

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#243

Post by mycall » Sun, Mar 01, 2026, 2:51 PM UTC

kir-gadjello wrote: post #128

Aurornis wrote: post #102

If you're new to this: All of the open source models are playing benchmark optimization games. Every new open weight model comes with promises of being as good as something SOTA from a few months ago then they always disappoint in actual use. I've been playing with Qwen3-Coder-Next and the Qwen3.5 models since they were each released. They are impressive, but they are not performing at Sonnet 4.5 level in my experien…

Respectfully, from my experience and a few billions of tokens consumed, some opensource models really are strong and useful. Specifically StepFun-3.5-flash https://github.com/stepfun-ai/Step-3.5-Flash I'm working on a pretty complex Rust codebase right now, with hundreds of integration tests and nontrivial concurrency, and stepfun powers through. I have no relation to stepfun, and I'm saying this purely from deep res…

TDD is really the delineation between being successful or not when using [local] LLMs.

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#244

Post by lurkshark » Sun, Mar 01, 2026, 3:14 PM UTC

amelius wrote: post #133

TrainedMonkey wrote: post #109

Earlier quoted context omitted.

No, what he is saying is that benchmarks are static and there is tremendous reputational and financial pressure to make benchmark number go up. So you add specific problems to training data... The result is that the model is smarter, but the benchmarks overstate the progress. Sure there are problem sets designed to be secret, but keeping secrets is hard given the fraction of planetary resources we are dedicating to m…

There should be a way to turn the questions we ask LLMs into benchmarks. That way, we can have a benchmark that is always up to date.

There are a few “updating” benchmarks out there. I periodically take a look at these two:

https://swe-rebench.com/

https://livebench.ai/

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#245

Post by SyneRyder » Sun, Mar 01, 2026, 3:20 PM UTC

8note wrote: post #98

lm28469 wrote: post #89

Earlier quoted context omitted.

Yes and Devstral 2 24b q4 is supposed to be 90% as good but it can't even reliably write to a file on my machine. There are the benchmarks, the promises, and what everybody can try at home

maybe a harness problem?

Having tried the Mistral Vibe harness that was supposedly designed for Devstral, that thing is abysmal. I feel sorry for whatever they did to that model, it didn't deserve it.

The thing I most noticed was asking it for help with configuring local MCP servers in Mistral Vibe - something it supports, it literally shows how many MCP servers are connected on the startup screen - it then begins scanning my local machine for servers running "MineCraft Protocol".

I want Mistral to do well, and I use their Voxtral Transcribe 2, that one has been useful. I'd even like a well made Mistral Vibe (c'mon, "oui oui baguette" is a hilarious replacement for "thinking"). But Mistral are so far behind, and they don't seem to even know or accept that they are.

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#246

Post by airstrike » Sun, Mar 01, 2026, 3:22 PM UTC

nl wrote: post #166

airstrike wrote: post #151

Earlier quoted context omitted.

is your prompt literally 1-sentence? if so, a better approach would be to ask it to first plan that entire task and give it some specific guidance then once it has the plan, ask it to execute it, preferably by letting it call other subagents that take care of different phases of the implementation while the main loop just merges those worktrees back it's how you should be using claude code too, btw

Claude Sonnet can easily one-shot that without specifically asking for plan first.

I believe you, but performance on 10-word prompts is pretty useless as a metric

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#247

Post by lubitelpospat » Sun, Mar 01, 2026, 3:23 PM UTC

carlio wrote: post #190

lubitelpospat wrote: post #188

All right guys, this is your time - what consumer device do you use for local LLM inference? GPU poor answers only

An AMD AI max+ 395 - I use the one from frame.work ( https://frame.work/de/en/desktop ) with 128GB unified RAM and it can run a 120b model (gpt-oss:120b) just fine. See Wendel's review here - https://www.youtube.com/watch?v=L-xgMQ-7lW0 There are other mini-pc manufacturers, the mainboard is the important part.

Wow, that's quite beefy.

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#248

Post by mstaoru » Sun, Mar 01, 2026, 3:32 PM UTC

zozbot234 wrote: post #55

mstaoru wrote: post #53

I periodically try to run these models on my MBP M3 Max 128G (which I bought with a mind to run local AI). I have a certain deep research question (in a field that is deeply familiar to me) that I ask when I want to gauge model's knowledge. So far Opus 4.6 and Gemini Pro are very satisfactory, producing great answers fairly fast. Gemini is very fast at 30-50 sec, Opus is very detailed and comes at about 2-3 minutes.…

Running local AI models on a laptop is a weird choice. The Mini and especially the Studio form factor will have better cooling, lower prices for comparable specs and a much higher ceiling in performance and memory capacity.

So it's back to the original question, why spend $5-10k on the Studio, when it will still be 10x slower and half the intelligence vs. $20 Sonnet?.. What is the point (besides privacy) to use local models now for coding?

PS: I can understand that isolated "valuable" problems like sorting photo collection or feeding a cat via ESPHome can be solved with local models.

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#249

Post by rahimnathwani » Sun, Mar 01, 2026, 3:34 PM UTC

suprjami wrote: post #18

rahimnathwani wrote: post #15

Earlier quoted context omitted.

There seem to be a lot of different Q4s of this model: https://www.reddit.com/r/LocalLLaMA/s/kHUnFWZXom I'm curious which one you're using.

Unsloth Dynamic. Don't bother with anything else.

For anyone else trying to run this on a Mac with 32GB unified RAM, this is what worked for me:

First, make sure enough memory is allocated to the gpu:

  sudo sysctl -w iogpu.wired_limit_mb=24000

Then run llama.cpp but reduce RAM needs by limiting the context window and turning off vision support. (And turn off reasoning for now as it's not needed for simple queries.)

  llama-server \
    -hf unsloth/Qwen3.5-35B-A3B-GGUF:UD-Q4_K_XL \
    --jinja \
    --no-mmproj \
    --no-warmup \
    -np 1 \
    -c 8192 \
    -b 512 \
    --chat-template-kwargs '{"enable_thinking": false}'

You can also enable/disable thinking on a per-request basis:

  curl 'http://localhost:8080/v1/chat/completions' \
  --data-raw '{"messages":[{"role":"user","content":"hello"}],"stream":false,"return_progress":false,"reasoning_format":"auto","temperature":0.8,"max_tokens":-1,"dynatemp_range":0,"dynatemp_exponent":1,"top_k":40,"top_p":0.95,"min_p":0.05,"xtc_probability":0,"xtc_threshold":0.1,"typ_p":1,"repeat_last_n":64,"repeat_penalty":1,"presence_penalty":0,"frequency_penalty":0,"dry_multiplier":0,"dry_base":1.75,"dry_allowed_length":2,"dry_penalty_last_n":-1,"samplers":["penalties","dry","top_n_sigma","top_k","typ_p","top_p","min_p","xtc","temperature"],"chat_template_kwargs": { "enable_thinking": true }}'|jq .

If anyone has any better suggestions, please comment :)

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#250

Post by regularfry » Sun, Mar 01, 2026, 3:38 PM UTC

827a wrote: post #175

vlovich123 wrote: post #79

Earlier quoted context omitted.

The hardware difference explains runtime performance differences, not task performance. Speculation is that the frontier models are all below 200B parameters but a 2x size difference wouldn’t fully explain task performance differences

He's running a 35B parameter model. Frontier models are well over a trillion parameters at this point. Parameters = smarts. There are 1T+ open source models (e.g. GLM5), and they're actually getting to the point of being comparable with the closed source models; but you cannot remotely run them on any hardware available to us. Core speed/count and memory bandwidth determines your performance. Memory size determines y…

The architecture is also important: there's a trade-off for MoE. There used to be a rough rule of thumb that a 35bxa3b model would be equivalent in smarts to an 11b dense model, give or take, but that's not been accurate for a while.