Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen3.6-35B-A3B: Agentic coding power, now open to all

qwen.ai

Post reply on HN
Print view View original

201–210 of 563 posts

Re: Qwen3.6-35B-A3B: Agentic coding power, now open to all

#201

Post by danielhanchen »

rohansood15 wrote: post #192
danielhanchen wrote: post #161

Earlier quoted context omitted.

1. Gemma-4 we re-uploaded 4 times - 3 times were 10-20 llama.cpp bug fixes - we had to notify people to upload the correct ones. The 4th is an official Gemma chat template improvement from Google themselves. 2. Qwen3.5 - we shared our 7TB research artifacts showing which layers not to quantize - all provider's quants were under optimized, not broken - ssm_out and ssm_* tensors were the issue - we're now the best in t…

Thanks for all the amazing work Daniel. I remember you guys being late to OH because you were working on weights released the night before - and it's great to see you guys keep up the speed!

Oh thanks haha :) We try our best to get model releases out the door! :) Hope you're doing great!

Re: Qwen3.6-35B-A3B: Agentic coding power, now open to all

#202

Post by danielhanchen »

bildung wrote: post #182
danielhanchen wrote: post #171

Earlier quoted context omitted.

No Bartowski's are more affected - (38% NaN) than ours (22%) - for MiniMax 2.7 see https://www.reddit.com/r/LocalLLaMA/comments/1slk4di/minimax... We already fixed ours. Bart hasn't yet but is still working on it following our findings. blk.61.ffn_down_exps in Q4_K or Q5_K failed - it must be in Q6_K otherwise it overflows. For the others, yes layers in some precision don't work. For eg Qwen3.5 ssm_out must be minimu…

Thanks again, TIL

Thanks!

Re: Qwen3.6-35B-A3B: Agentic coding power, now open to all

#203

Post by Glemllksdf »

Aurornis wrote: post #173
WithinReason wrote: post #106

Earlier quoted context omitted.

It's on the page: Precision Quantization Tag File Size 1-bit UD-IQ1_M 10 GB 2-bit UD-IQ2_XXS 10.8 GB UD-Q2_K_XL 12.3 GB 3-bit UD-IQ3_XXS 13.2 GB UD-Q3_K_XL 16.8 GB 4-bit UD-IQ4_XS 17.7 GB UD-Q4_K_XL 22.4 GB 5-bit UD-Q5_K_XL 26.6 GB 16-bit BF16 69.4 GB

Additional VRAM is needed for context. This model is a MoE model with only 3B active parameters per expert which works well with partial CPU offload. So in practice you can run the -A(N)B models on systems that have a little less VRAM than you need. The more you offload to the CPU the slower it becomes though.

Isn't that some kind of gambling if you offload random experts onto the CPU?

Or is it only layers but that would affect all Experts?

Re: Qwen3.6-35B-A3B: Agentic coding power, now open to all

#204

Post by bigyabai »

zozbot234 wrote: post #193
bigyabai wrote: post #186

Earlier quoted context omitted.

taps the sign Unified Memory Is A Marketing Gimmeck. Industrial-Scale Inference Servers Do Not Use It.

Industrial Scale Inference is moving towards LPDDR memory (alongside HBM), which is essentially what "Unified Memory" is.

LPDDR is LPDDR. There's nothing "unified" about it architecturally.

Re: Qwen3.6-35B-A3B: Agentic coding power, now open to all

#205

Post by Glemllksdf »

cjbgkagh wrote: post #125
utilize1808 wrote: post #111

Earlier quoted context omitted.

Obviously going to depend on your definition of "decent". My impression so far is that you will need between 90GB to 100GB of memory to run medium sized (31B dense or ~110B MoE) models with some quantization enabled.

I’m running Gemma4 31B (Q8) on my 2 4090s (48GB) with no problem.

I have the same setup but tried paperclip ai with it and it seems to me that either i'm unable to setup it properly or multiply agents struggle with this setup. Especially as it seems that paperclip ai and opencode (used for connection) is blowing up the context to 20-30k

Any tips around your setup running this?

I use lmstudio with default settings and prioritization instead of split.

Re: Qwen3.6-35B-A3B: Agentic coding power, now open to all

#206

Post by sowbug »

danielhanchen wrote: post #198
Aurornis wrote: post #176

Earlier quoted context omitted.

Unsloth is great for uploading quants quickly to experiment with, but everyone should know that they almost always revise their quants after testing. If you download the release day quants with a tool that doesn’t automatically check HF for new versions you should check back again in a week to look for updated versions. Some times the launch day quantizations have major problems which leads to early adopters dismissi…

We re-uploaded Gemma4 4 times - 3 times were due to 20 llama.cpp bug fixes, which we helped solve some as well. The 4th is an official Gemma chat template improvement from Google themselves, so these are out of our hands. All providers had to re-fix their uploads, so not just us. For MiniMax 2.7 - there were NaNs, but it wasn't just ours - all quant providers had it - we identified 38% of bartowski's had NaNs. Ours w…

Please publish sha256sums of the merged GGUFs in the model descriptions. Otherwise it's hard to tell if the version we have is the latest.

Re: Qwen3.6-35B-A3B: Agentic coding power, now open to all

#207

Post by sander1095 »

homebrewer wrote: post #18

Already quantized/converted into a sane format by Unsloth: https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF

I sense that I don't really understand enough of your comment to know why this is important. I hope you can explain some things to me:

- Why is Qwen's default "quantization" setup "bad" - Who is Unsloth? - Why is his format better? What gains does a better format give? What are the downsides of a bad format? - What is quantization? Granted, I can look up this myself, but I thought I'd ask for the full picture for other readers.

Re: Qwen3.6-35B-A3B: Agentic coding power, now open to all

#209

Post by danielhanchen »

sowbug wrote: post #206
danielhanchen wrote: post #198

Earlier quoted context omitted.

We re-uploaded Gemma4 4 times - 3 times were due to 20 llama.cpp bug fixes, which we helped solve some as well. The 4th is an official Gemma chat template improvement from Google themselves, so these are out of our hands. All providers had to re-fix their uploads, so not just us. For MiniMax 2.7 - there were NaNs, but it wasn't just ours - all quant providers had it - we identified 38% of bartowski's had NaNs. Ours w…

Please publish sha256sums of the merged GGUFs in the model descriptions. Otherwise it's hard to tell if the version we have is the latest.

Yep we can do that probs add a table - in general be post in discussions of model pages - for eg https://huggingface.co/unsloth/MiniMax-M2.7-GGUF/discussions...

HF also provides SHA256 for eg https://huggingface.co/unsloth/MiniMax-M2.7-GGUF/blob/main/U... is 92986e39a0c0b5f12c2c9b6a811dad59e3317caaf1b7ad5c7f0d7d12abc4a6e8

But agreed it's probs better to place them in a table

Re: Qwen3.6-35B-A3B: Agentic coding power, now open to all

#210

Post by wrxd »

fooblaster wrote: post #5

Honestly, this is the AI software I actually look forward to seeing. No hype about it being too dangerous to release. No IPO pumping hype. No subscription fees. I am so pumped to try this!

Same here. I really hope in a near future local model will be good enough and hardware fast enough to run them to become viable for most use cases
Post reply on HN