Our shop cannot use cloud models for sensitive data and code. For shops like ours, we continue to be impressed and appreciative of the progress in open-source / self-hosted models.
Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers
231–240 of 286 posts
Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers
#232Earlier quoted context omitted.
Qwen has been broadly aligned to give positive messages about China in English. https://chinamediaproject.org/2026/02/09/tokens-of-ai-bias/ An Analysis of Chinese LLM Censorship and Bias with Qwen 2 Instruct https://huggingface.co/blog/leonardlin/chinese-llm-censorshi...
Impressive, very nice, now let's see what would be the odds that the US models developed in SV are also highly positive about Californian and Democrats politics.
Whataboutism
https://en.wikipedia.org/wiki/Whataboutism
Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers
#233I periodically try to run these models on my MBP M3 Max 128G (which I bought with a mind to run local AI). I have a certain deep research question (in a field that is deeply familiar to me) that I ask when I want to gauge model's knowledge. So far Opus 4.6 and Gemini Pro are very satisfactory, producing great answers fairly fast. Gemini is very fast at 30-50 sec, Opus is very detailed and comes at about 2-3 minutes.…
I've seen reports of qwen3.5-35b-a3b spending a ton of time reasoning if the context window is nearly empty-- supposedly it reasons less if you provide a long system prompt or some file contents, like if you use it in a coding agent. I'm too GPU-poor to run it, but r/LocalLLaMa is full of people using it.
There's definitely something wrong with the thinking mode on this one. I wouldn't be surprised if it gets fixed, either by qwen themselves or with a fine-tune.
Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers
#234If you're new to this: All of the open source models are playing benchmark optimization games. Every new open weight model comes with promises of being as good as something SOTA from a few months ago then they always disappoint in actual use. I've been playing with Qwen3-Coder-Next and the Qwen3.5 models since they were each released. They are impressive, but they are not performing at Sonnet 4.5 level in my experien…
I've been trying to get these things to local host and use tools. Am I right in understanding that it's impossible for these things to use tools from within llama.cpp? Do I need another "thing" to run the models? What exactly is the mechanism by which the models became aware that they're somewhere where they have tools availbale? So many questions...
Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers
#235Earlier quoted context omitted.
Your Gemini or Opus question got send to a Texas datacenter where it got queued and processed by a subunit of 80 h200 140gb 1000w cards running a many billion or trillion parameter model. It took less that 200ms to process a single request. Your Claude cliënt decided to spawn 30 sub agents and iterated in a total of 90 requests totalling about 45000ms. Now compare that to your 100b transistor cpu doing something simi…
Right, it was more of a rhetorical question :) With my point being - how are these local models really useful to me now? Is the Only Way ™ to sell my house and build a 8x5090 monster?.. How does that compare to $20/month Opus? (Privacy aside.) The second order thought from this is... will we get a value-based price leveling soon? If the alternative to a hosted LLM is to build $10-20k+ machine with $500+ monthly energ…
Looked at from the other end of the telescope, the other factor is how fast low-end local models can gain capability. This 35b model is absolutely fine on a 4090 in a machine that was about £3000 when I bought it three years ago. Where will what you can run on a 4090, or a 5090, be in six months? That's the interesting question, but we're already well past the point where the uses to which you will be able to put a local model dramatically increase within the depreciation lifespan of the hardware.
Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers
#236I periodically try to run these models on my MBP M3 Max 128G (which I bought with a mind to run local AI). I have a certain deep research question (in a field that is deeply familiar to me) that I ask when I want to gauge model's knowledge. So far Opus 4.6 and Gemini Pro are very satisfactory, producing great answers fairly fast. Gemini is very fast at 30-50 sec, Opus is very detailed and comes at about 2-3 minutes.…
Try the 27B dense model. It will likely do much better than the 35b MoE with only 3B active experts. Also, performance on research-y questions isn't always a good indicator of how the model will do for code generation or agent orchestration.
Currently sat waiting for the unsloth fixed quants to drop, but I'm on the edge of my seat for this.
Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers
#237I am a total neophyte when it comes to LLMs, and only recently started poking around into the internals of them. The first thing that struck me was that float32 dimensions seemed very generous . I then discovered what quantization is by reading a blog post about binary quantization. That seemed too good to be true. I asked Claude to design an analysis assessing the fidelity of 1, 2, 4, and 8 bit quantization. Claude…
There's also work on ternary models that's quite interesting, because the arithmetic operations are super fast and they're extremely cache efficient. Well worth looking into if that's the sort of thing that interests you.
Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers
#238Can someone who has done this, simplify and say what specs we need on a `local computer` to run and test this, with a reasonable speed? Excluding MBP M5 128GB.
I've got the unsloth q4_K_XL 35b running in llama.cpp on an i9/64G/4090 machine doing double-digit tokens per second with a 90k+ token context window available. The model's completely in VRAM.
Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers
#239Re: Qwen3.5 122B and 35B models offer Sonnet 4.5 performance on local computers
#240Are there any non-Chinese open models that offer comparable performance?
It's not only "non-Chinese" to think about here. There's nobody really touching Qwen in the single-GPU size class and there hasn't been for a couple of generations.