Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

venturebeat.com

Post reply on HN
Print view View original

181–190 of 286 posts

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#181

Post by xtn »

mstaoru wrote: post #53

I periodically try to run these models on my MBP M3 Max 128G (which I bought with a mind to run local AI). I have a certain deep research question (in a field that is deeply familiar to me) that I ask when I want to gauge model's knowledge. So far Opus 4.6 and Gemini Pro are very satisfactory, producing great answers fairly fast. Gemini is very fast at 30-50 sec, Opus is very detailed and comes at about 2-3 minutes.…

I think knowledge of frontier research certainly scale with number of parameters. Also, US labs can pay more money to have researchers provide training data on these frontier research areas.

On the other hand, if indeed open source models and Macbooks can be as powerful as those SOTA models from Google, etc, then stock prices of many companies would already collapsed.

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#183

Post by throwdbaaway »

DetroitThrow wrote: post #144
CharlesW wrote: post #71

Earlier quoted context omitted.

Not sure what this means, but as a marketing person myself, here's what happened: One day, an Anthropican involved in the Haiku 4.5 launch shrugged, weighed the odds of getting spanked for equating "extended thinking" with "reasoning", and then used Claude to generate copy declaring that. It's not rocket surgery!

It's mainly that people on here, regardless of profession, speak incorrectly but confidentally about things that could be easily verified with a Google search or basic familiarity with the thing in question. Haiku 4.5 is a reasoning model, regardless of whatever hallucination you read. Being a hybrid reasoning model means that, depending on the complexity of the question and whether you explicitly enable reasoning (t…

We are all reasonable people here, and while you are (mostly) correct, I think we can all agree that Anthropic documentation sucks. If I have to infer from the doc:

* Haiku 4.5 by default doesn't think, i.e. it has a default thinking budget of 0.

* By setting a non-zero thinking budget, Haiku 4.5 can think. My guess is that Claude Code may set this differently for different tasks, e.g. thinking for Explore, no thinking for Compact.

* This hybrid thinking is different from the adaptive thinking introduced in Opus 4.6, which when enabled, can automatically adjust the thinking level based on task difficulty.

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#184

Post by ekianjo »

Aurornis wrote: post #102

If you're new to this: All of the open source models are playing benchmark optimization games. Every new open weight model comes with promises of being as good as something SOTA from a few months ago then they always disappoint in actual use. I've been playing with Qwen3-Coder-Next and the Qwen3.5 models since they were each released. They are impressive, but they are not performing at Sonnet 4.5 level in my experien…

> That said, they are impressive for open source models.

there is nothing open "source" about them. They are open weights, that's all.

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#185

Post by throwdbaaway »

CamperBob2 wrote: post #142
PunchyHamster wrote: post #76

Earlier quoted context omitted.

good to know, thanks. I just ran ollama with qwen3.5:27b. Currently it's stuck on picking format Let's write. Wait, I'll write the response. Wait, I'll check if I should use a table. No, text is fine. Okay. Let's write. Wait, I'll write the response. Wait, I'll check if I should use a bullet list. No, just lines. Okay. Let's write. Wait, I'll write the response. Wait, I'll check if I should use a numbered list. No, l…

What quant? I just ran Repeat the word "potato" 100 times, numbered and it worked fine, taking 44 seconds at 24 tokens/second. Command line: llama-server ^ --model Qwen3.5-27B-BF16-00001-of-00002.gguf ^ --mmproj mmproj-BF16.gguf ^ --fit on ^ --host 127.0.0.1 ^ --port 2080 ^ --temp 0.8 ^ --top-p 0.95 ^ --top-k 20 ^ --min-p 0.00 ^ --presence_penalty 1.5 ^ --repeat_penalty 1.1 ^ --no-mmap ^ --no-warmup The repeat and/or…

I don't quite get the low temperature coupled with the high penalty. We get thinking loop due to low temperature, and we then counter it with high penalty. That seems backward.

For Qwen3.5 27B, I got good result with --temp 1.0 --top-p 1.0 --top-k 40 --min-p 0.2, without penalty. It allows the model to explore (temp, top-p, top-k) without going off the rail (min-p) during reasoning. No loop so far.

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#186

Post by copperx »

kir-gadjello wrote: post #128
Aurornis wrote: post #102

If you're new to this: All of the open source models are playing benchmark optimization games. Every new open weight model comes with promises of being as good as something SOTA from a few months ago then they always disappoint in actual use. I've been playing with Qwen3-Coder-Next and the Qwen3.5 models since they were each released. They are impressive, but they are not performing at Sonnet 4.5 level in my experien…

Respectfully, from my experience and a few billions of tokens consumed, some opensource models really are strong and useful. Specifically StepFun-3.5-flash https://github.com/stepfun-ai/Step-3.5-Flash I'm working on a pretty complex Rust codebase right now, with hundreds of integration tests and nontrivial concurrency, and stepfun powers through. I have no relation to stepfun, and I'm saying this purely from deep res…

Are you using stepfun mostly because it's free, or is it better than other models at some things?

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#187

Post by chr15m »

shell0x wrote: post #172
xenospn wrote: post #4

Are there any non-Chinese open models that offer comparable performance?

What's the problem with Chinese models? The models are already open which makes them more trustworthy than the American closed models.

They are trained to respond to certain topics in a way that does not align with real world evidence. Pretty much the opposite of what you want in such a tool.

This is trivial to test and verify yourself. Just pick any topic you think has a chance of being censored. You can do the same on American models and compare results.

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#189

Post by CamperBob2 »

throwdbaaway wrote: post #185
CamperBob2 wrote: post #142

Earlier quoted context omitted.

What quant? I just ran Repeat the word "potato" 100 times, numbered and it worked fine, taking 44 seconds at 24 tokens/second. Command line: llama-server ^ --model Qwen3.5-27B-BF16-00001-of-00002.gguf ^ --mmproj mmproj-BF16.gguf ^ --fit on ^ --host 127.0.0.1 ^ --port 2080 ^ --temp 0.8 ^ --top-p 0.95 ^ --top-k 20 ^ --min-p 0.00 ^ --presence_penalty 1.5 ^ --repeat_penalty 1.1 ^ --no-mmap ^ --no-warmup The repeat and/or…

I don't quite get the low temperature coupled with the high penalty. We get thinking loop due to low temperature, and we then counter it with high penalty. That seems backward. For Qwen3.5 27B, I got good result with --temp 1.0 --top-p 1.0 --top-k 40 --min-p 0.2, without penalty. It allows the model to explore (temp, top-p, top-k) without going off the rail (min-p) during reasoning. No loop so far.

The guidelines are a little hard to interpret. At https://huggingface.co/Qwen/Qwen3.5-27B Qwen says to use temp 0.6, pres 0.0, rep 1.0 for "thinking mode for precise coding tasks" and temp 1.0, pres 1.5, rep 1.0 for "thinking mode for general tasks." Those parameters are just swinging wildly all over the place, and I don't know if printing potato 100 times is considered to be more like a "precise coding task" or a "general task."

When setting up the batch file for some previous tests, I decided to split the difference between 0.6 and 1.0 for temperature and use the larger recommended values for presence and repetition. For this prompt, it probably isn't a good idea to discourage repetition, I guess. But keeping the existing parameters worked well enough, so I didn't mess with them.

Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers

#190

Post by carlio »

lubitelpospat wrote: post #188

All right guys, this is your time - what consumer device do you use for local LLM inference? GPU poor answers only

An AMD AI max+ 395 - I use the one from frame.work (https://frame.work/de/en/desktop) with 128GB unified RAM and it can run a 120b model (gpt-oss:120b) just fine.

See Wendel's review here - https://www.youtube.com/watch?v=L-xgMQ-7lW0

There are other mini-pc manufacturers, the mainboard is the important part.

Post reply on HN