Anybody use these instead of codex or claude code? Thoughts in comparison?
benchmarks dont really help me so much
Qwen3.6-35B-A3B: Agentic coding power, now open to all
191–200 of 563 posts
Re: Qwen3.6-35B-A3B: Agentic coding power, now open to all
#192Earlier quoted context omitted.
Bad QA :/ They had a bunch of broken quantizations in the last releases
1. Gemma-4 we re-uploaded 4 times - 3 times were 10-20 llama.cpp bug fixes - we had to notify people to upload the correct ones. The 4th is an official Gemma chat template improvement from Google themselves. 2. Qwen3.5 - we shared our 7TB research artifacts showing which layers not to quantize - all provider's quants were under optimized, not broken - ssm_out and ssm_* tensors were the issue - we're now the best in t…
Thanks for all the amazing work Daniel. I remember you guys being late to OH because you were working on weights released the night before - and it's great to see you guys keep up the speed!
Re: Qwen3.6-35B-A3B: Agentic coding power, now open to all
#193More benchmaxxing I see. Too bad there’s no rig with 256gb unified ram for under $1000
taps the sign Unified Memory Is A Marketing Gimmeck. Industrial-Scale Inference Servers Do Not Use It.
Industrial Scale Inference is moving towards LPDDR memory (alongside HBM), which is essentially what "Unified Memory" is.
Re: Qwen3.6-35B-A3B: Agentic coding power, now open to all
#194I'm broadly curious how people are using these local models. Literally, how are they attaching harnesses to this and finding more value than just renting tokens from Anthropic of OpenAI?
It’s easy to find a combination of llama.cpp and a coding tool like OpenCode for these. Asking an LLM for help setting it up can work well if you don’t want to find a guide yourself.
> and finding more value than just renting tokens from Anthropic of OpenAI?
Buying hardware to run these models is not cost effective. I do it for fun for small tasks but I have no illusions that I’m getting anything superior to hosted models. They can be useful for small tasks like codebase exploration or writing simple single use tools when you don’t want to consume more of your 5-hour token budget though.
Re: Qwen3.6-35B-A3B: Agentic coding power, now open to all
#195Earlier quoted context omitted.
32 GiB of VRAM is possible to acquire for less than $1000 if you go for the Arc Pro B70. I have two of them. The tokens/sec is nowhere near AMD or NVIDIA high end, but its unexpectedly kind of decent to use. (I probably need to figure out vLLM though as it doesn't seem like llama.cpp is able to do them justice even seemingly with split mode = row. But still, 30t/s on Gemma 4 (on 26B MoE, not dense) is pretty usable,…
New versions of llama.cpp have experimental split-tensor parallelism, but it really only helps with slow compute and a very fast interconnect, which doesn't describe many consumer-grade systems. For most users, pipeline parallelism will be their best bet for making use of multi-GPU setups.
Yeah, I was doing split tensor and it seemed like a wash. The Arc B70s are not huge on compute.
Right now I'm only able to run them in PCI-e 5.0 x8 which might not be sufficient. But, a cheap older Xeon or TR seems silly since PCI-e 4.0 x16 isn't theoretically more bandwidth than PCI-e 5.0 x8. So it seems like if that is really still bottlenecked, I'll just have to bite the bullet and set up a modern HEDT build. With RAM prices... I am not sure there is a world where it could ever be worth it. At that point, seems like you may as well go for an obscenely priced NVIDIA or AMD datacenter card instead and retrofit it with consumer friendly thermal solutions. So... I'm definitely a bit conflicted.
I do like the Arc Pro B70 so far. Its not a performance monster, but it's quiet and relatively low power, and I haven't run into any instability. (The AMDGPU drivers have made amazing strides, but... The stability is not legendary. :)
I'll have to do a bit of analysis and make sure there really is an interconnect bottleneck first, versus a PEBKAC. Could be dropping more lanes than expected for one reason or another too.
Re: Qwen3.6-35B-A3B: Agentic coding power, now open to all
#196I'm broadly curious how people are using these local models. Literally, how are they attaching harnesses to this and finding more value than just renting tokens from Anthropic of OpenAI?
Some tasks don’t require SOTA models. For translating small texts I use Gemma 4 on my iPhone because it’s faster and better than Apple Translate or Google Translate and works offline. Also if you can break down certain tasks like JSON healing into small focused coding tasks then local models are useful
Re: Qwen3.6-35B-A3B: Agentic coding power, now open to all
#197Already quantized/converted into a sane format by Unsloth: https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF
Unsloth is great for uploading quants quickly to experiment with, but everyone should know that they almost always revise their quants after testing. If you download the release day quants with a tool that doesn’t automatically check HF for new versions you should check back again in a week to look for updated versions. Some times the launch day quantizations have major problems which leads to early adopters dismissi…
Not to mention that almost every model release has some (at least) minor issue in the prompt template and/or the runtime itself, so even if they (not talking unsloth specifically, in general) claim "Day 0 support", do pay extra attention to actual quality as it takes a week or two before issues been hammered out.
Re: Qwen3.6-35B-A3B: Agentic coding power, now open to all
#198Already quantized/converted into a sane format by Unsloth: https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF
Unsloth is great for uploading quants quickly to experiment with, but everyone should know that they almost always revise their quants after testing. If you download the release day quants with a tool that doesn’t automatically check HF for new versions you should check back again in a week to look for updated versions. Some times the launch day quantizations have major problems which leads to early adopters dismissi…
We re-uploaded Gemma4 4 times - 3 times were due to 20 llama.cpp bug fixes, which we helped solve some as well. The 4th is an official Gemma chat template improvement from Google themselves, so these are out of our hands. All providers had to re-fix their uploads, so not just us.
For MiniMax 2.7 - there were NaNs, but it wasn't just ours - all quant providers had it - we identified 38% of bartowski's had NaNs. Ours was 22%. We identified a fix, and have already fixed ours see https://www.reddit.com/r/LocalLLaMA/comments/1slk4di/minimax.... Bartowski has not, but is working on it. We share our investigations always.
For Qwen3.5 - we shared our 7TB research artifacts showing which layers not to quantize - all provider's quants were not optimal, not broken - ssm_out and ssm_* tensors were the issue - we're now the best in terms of KLD and disk space - see https://www.reddit.com/r/LocalLLaMA/comments/1rgel19/new_qwe...
On other fixes, we also fixed bugs in many OSS models like Gemma 1, Gemma 3, Llama chat template fixes, Mistral, and many more.
It might seem these issues are due to us, but it's because we publicize them and tell people to update. 95% of them are not related to us, but as good open source stewards, we should update everyone.
Re: Qwen3.6-35B-A3B: Agentic coding power, now open to all
#199Earlier quoted context omitted.
Unsloth is great for uploading quants quickly to experiment with, but everyone should know that they almost always revise their quants after testing. If you download the release day quants with a tool that doesn’t automatically check HF for new versions you should check back again in a week to look for updated versions. Some times the launch day quantizations have major problems which leads to early adopters dismissi…
Not to mention that almost every model release has some (at least) minor issue in the prompt template and/or the runtime itself, so even if they (not talking unsloth specifically, in general) claim "Day 0 support", do pay extra attention to actual quality as it takes a week or two before issues been hammered out.
Yes this is fair - we try our best to communicate issues - I think we're mostly the only ones doing the communication that model A or B has been fixed etc.
We try our best as model distributors to fix them on day 0 or 1, but 95% of issues aren't our issues - as you mentioned it's the chat template or runtime etc
Re: Qwen3.6-35B-A3B: Agentic coding power, now open to all
#200Earlier quoted context omitted.
New versions of llama.cpp have experimental split-tensor parallelism, but it really only helps with slow compute and a very fast interconnect, which doesn't describe many consumer-grade systems. For most users, pipeline parallelism will be their best bet for making use of multi-GPU setups.
Yeah, I was doing split tensor and it seemed like a wash. The Arc B70s are not huge on compute. Right now I'm only able to run them in PCI-e 5.0 x8 which might not be sufficient. But, a cheap older Xeon or TR seems silly since PCI-e 4.0 x16 isn't theoretically more bandwidth than PCI-e 5.0 x8. So it seems like if that is really still bottlenecked, I'll just have to bite the bullet and set up a modern HEDT build. With…
You could fit your HEDT with minimum RAM and a combination of Optane storage (for swapping system RAM with minimum wear) and fast NAND (for offloading large read-only data). If you have abundant physical PCIe slots it ought to be feasible.