Live data from Hacker News

Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

venturebeat.com

151–160 of 286 posts

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#151
post #68

Smells like hyperbole. A lot of people making such claims don’t seem to have continued real world experience with these models or seem to have very weird standards for what they consider usable. Up until relatively recently, while people had already long been making these claims, it came with the asterisks of „oh, but you can’t practically use more than a few K tokens of context“.

"Create a single page web app scientific RPN calculator" Qwen 3.5 122b/a10b (at q3 using unsloth's dynamic quant) is so far the first model I've tried locally that gets a really usable RPN calculator app. Other models (even larger ones that I can run on my Strix Halo box) tend to either not implement the stack right, have non-functional operation buttons, or most commonly the keypad looks like a Picasso painting (i.e…

is your prompt literally 1-sentence?

if so, a better approach would be to ask it to first plan that entire task and give it some specific guidance

then once it has the plan, ask it to execute it, preferably by letting it call other subagents that take care of different phases of the implementation while the main loop just merges those worktrees back

it's how you should be using claude code too, btw

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#152

Earlier quoted context omitted.

Thats cool. I do wonder where that extra acuity you get from 1% more shows up in practice. I hate how I have basically no way to intuitively tell that because of how much of a black box the system is

Well why would Claude know any of this? Obviously it's the wrong criteria. If you have your own dataset to benchmark, created your own calibration for quantization with it. Scientifically, you wouldn't really believe in the whole process of gradient descent if you didn't think tiny differences in these values matter. So...

I think you might be answering to a different person or misunderstanding what I said but you are right that just as I don’t have an intuition for where the acuity shows up in the corpus, I don’t think Claude does either

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#153

I am a total neophyte when it comes to LLMs, and only recently started poking around into the internals of them. The first thing that struck me was that float32 dimensions seemed very generous . I then discovered what quantization is by reading a blog post about binary quantization. That seemed too good to be true. I asked Claude to design an analysis assessing the fidelity of 1, 2, 4, and 8 bit quantization. Claude…

4-bit quantization on newer nvidia hardware is being supported in training as well these days. I believe the gpt-oss models were trained natively in MXFP4, which is a 4-bit floating point / e2m1 (2-exponent, 1 bit mantissa, 1 bit sign).

It doesn't seem terribly common yet though. I think it is challenging to keep it stable.

[1] https://www.opencompute.org/blog/amd-arm-intel-meta-microsof...

[2] https://www.opencompute.org/documents/ocp-microscaling-forma...

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#154
post #58

I recently wrote a guide on getting: - llama.cpp - OpenCode - Qwen3-Coder-30B-A3B-Instruct in GGUF format (Q4_K_M quantization) working on a M1 MacBook Pro (e.g. using brew). It was bit finicky to get all of the pieces together so hopefully this can be used with these newer models. https://gist.github.com/alexpotato/5b76989c24593962898294038...

On my 32GB Ryzen desktop (recently upgraded from 16GB before the RAM prices went up another +40%), did the same setup of llama.cpp (with Vulkan extra steps) and also converged on Qwen3-Coder-30B-A3B-Instruct (also Q4_K_M quantization) On the model choice: I've tried latest gemma, ministral, and a bunch of others. But qwen was definitely the most impressive (and much faster inference thanks to MoE architecture), so ca…

Quant choice depends on your vram, use case, need for speed, etc. For coding I would not go below Q4_K_M (though for Q4, unsloth XL or ik_llama IQ quants are usually better at the same size). Preferably Q5 or even Q6.

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#155
post #72

Earlier quoted context omitted.

I think you could look into Minstral. There's also GPT-OSS but I'm not sure how well it stacks up. What's your problem with Chinese LLMs?

it’s not obvious to you why someone would want to avoid models created by our enemies?

No, explain it to me. GPT-OSS is one of the most heavily-censored models on the internet, what's the point of buying local if it's crap?

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#156

I am a total neophyte when it comes to LLMs, and only recently started poking around into the internals of them. The first thing that struck me was that float32 dimensions seemed very generous . I then discovered what quantization is by reading a blog post about binary quantization. That seemed too good to be true. I asked Claude to design an analysis assessing the fidelity of 1, 2, 4, and 8 bit quantization. Claude…

Mind sharing any resources? I've been thinking about trying to understand them better myself.

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#158

Earlier quoted context omitted.

Respectfully, from my experience and a few billions of tokens consumed, some opensource models really are strong and useful. Specifically StepFun-3.5-flash https://github.com/stepfun-ai/Step-3.5-Flash I'm working on a pretty complex Rust codebase right now, with hundreds of integration tests and nontrivial concurrency, and stepfun powers through. I have no relation to stepfun, and I'm saying this purely from deep res…

What are you running that model on?

A 3 bit quant will run on a 128gb MacBook Pro, it works pretty well.

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#159

If you're new to this: All of the open source models are playing benchmark optimization games. Every new open weight model comes with promises of being as good as something SOTA from a few months ago then they always disappoint in actual use. I've been playing with Qwen3-Coder-Next and the Qwen3.5 models since they were each released. They are impressive, but they are not performing at Sonnet 4.5 level in my experien…

I'm using Qwen 3.5 27b on my 4090 and let me tell you. This is the first time I am seriously blown away by coding performance on a local model. They are almost always unusable. Not this time though...

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#160

Earlier quoted context omitted.

What are you running that model on?

A 3 bit quant will run on a 128gb MacBook Pro, it works pretty well.

A 3 bit quant is quite a lot weaker than the OpenRouter version the OP is using.
Post reply on HN