Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

venturebeat.com

Post reply on HN
Print view View original

131–140 of 286 posts

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#131

Post by kir-gadjello »

aappleby wrote: post #130
kir-gadjello wrote: post #128

Earlier quoted context omitted.

Respectfully, from my experience and a few billions of tokens consumed, some opensource models really are strong and useful. Specifically StepFun-3.5-flash https://github.com/stepfun-ai/Step-3.5-Flash I'm working on a pretty complex Rust codebase right now, with hundreds of integration tests and nontrivial concurrency, and stepfun powers through. I have no relation to stepfun, and I'm saying this purely from deep res…

What are you running that model on?

I just use openrouter, it's free for now. But I would pay 30-100$ to use it 24/7.

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#132

Post by cyberax »

dajonker wrote: post #19
erelong wrote: post #9

What kind of hardware does HN recommend or like to run these models?

Radeon R9700 with 32 GB VRAM is relatively affordable for the amount of RAM and with llama.cpp it runs fast enough for most things. These are workstation cards with blower fans and they are LOUD. Otherwise if you have the money to burn get a 5090 for speeeed and relatively low noise, especially if you limit power usage.

I have a pair of Radeon AI PRO R9700 with 32Gb, and so far they have been a pleasure to use. Drivers work out-of-the-box, and they are completely quiet when unused. They are capped at 300W power, so even at 100% utilization they are not too loud.

I was thinking about adding after-market liquid cooling for them, but they're fine without it.

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#133

Post by amelius »

TrainedMonkey wrote: post #109
amelius wrote: post #104

Earlier quoted context omitted.

Are you saying that the benchmarks are flawed? And could quantization maybe partially explain the worse than expected results?

No, what he is saying is that benchmarks are static and there is tremendous reputational and financial pressure to make benchmark number go up. So you add specific problems to training data... The result is that the model is smarter, but the benchmarks overstate the progress. Sure there are problem sets designed to be secret, but keeping secrets is hard given the fraction of planetary resources we are dedicating to m…

There should be a way to turn the questions we ask LLMs into benchmarks.

That way, we can have a benchmark that is always up to date.

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#134

Post by syntaxing »

A big part that a lot of local users forget is inference is hard. Maybe you have the wrong temperature. Maybe you have the wrong min P. Maybe you have the wrong template. Maybe the implementation in llama cpp has a bug. Maybe Q4 or even Q8 just won’t compare to BF16. Reality is, there’s so many knobs to LLM inferencing and any can make the experience worse. It’s not always the model’s fault.

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#135

Post by chaboud »

Aurornis wrote: post #102

If you're new to this: All of the open source models are playing benchmark optimization games. Every new open weight model comes with promises of being as good as something SOTA from a few months ago then they always disappoint in actual use. I've been playing with Qwen3-Coder-Next and the Qwen3.5 models since they were each released. They are impressive, but they are not performing at Sonnet 4.5 level in my experien…

"When a measure becomes a target, it ceases to be a good measure."

Goodhart's law shows up with people, in system design, in processor design, in education...

Models are going to be over-fit to the tests unless scruples or practical application realities intervene. It's a tale as old as machine learning.

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#136

Post by hsaliak »

No it does not. None of these models have the “depth” that the frontier models have across a variety of conversations, tasks and situations. Working with them is like playing snakes and ladders, you never know when it’s going to do something crazy and set you back.

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#137

Post by aappleby »

kir-gadjello wrote: post #131
aappleby wrote: post #130

Earlier quoted context omitted.

What are you running that model on?

I just use openrouter, it's free for now. But I would pay 30-100$ to use it 24/7.

Ah, I thought you meant you were running it locally.

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#138

Post by mark_l_watson »

johnnyApplePRNG wrote: post #20
mark_l_watson wrote: post #8

The new 35b model is great. That said, it has slight incompatibility's with Claude Code. It is very good for tool use.

Claude code is designed for anthropic models. Try it with opencode!

I will, right now.

EDIT: opencode was a bit slow with qwen3.5:35b using Ollama. Faster/nicer to use with Liquid lfm2:latest

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#140

Post by pamcake »

wolvoleo wrote: post #124
mstaoru wrote: post #53

I periodically try to run these models on my MBP M3 Max 128G (which I bought with a mind to run local AI). I have a certain deep research question (in a field that is deeply familiar to me) that I ask when I want to gauge model's knowledge. So far Opus 4.6 and Gemini Pro are very satisfactory, producing great answers fairly fast. Gemini is very fast at 30-50 sec, Opus is very detailed and comes at about 2-3 minutes.…

Well first of all you're running a long intense task on a thermally constrained machine. Your MacBook Pro is optimised for portability and battery life, not max performance under load. And apple's obsession with thinness overrules thermal performance for them. Short peaks will be ok but a 45 minute task will thoroughly saturate the cooling system. Even on servers this can happen. At work we have a 2U sized server wit…

To your point, one can get a great performance boost by propping the laptop onto a roost-like stand in front of a large fan. Nothing like a cooling system actually built for sustained load but still.
Post reply on HN