Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen3.6-35B-A3B: Agentic coding power, now open to all

qwen.ai

Post reply on HN
Print view View original

421–430 of 563 posts

Re: Qwen3.6-35B-A3B: Agentic coding power, now open to all

#421

Post by zozbot234 »

FuckButtons wrote: post #414
zozbot234 wrote: post #236

Earlier quoted context omitted.

You should be able to save a lot on prefill by stashing KV-cache shared prefixes (since KV-cache for plain transformers is an append-only structure) to near-line bulk storage and fetching them in as needed. Not sure why local AI engines don't do this already since it's a natural extension of session save/restore and what's usually called prompt caching.

if I understand you correctly, this is essentially what vllm does with their paged cache, if I’ve misunderstood I apologize.

Paged Attention is more of a low-level building block, aimed initially at avoiding duplication of shared KV-cache prefixes in large-batch inference. But you're right that it's quite related. The llama.cpp folks are still thinking about it, per a recent discussion from that project: https://github.com/ggml-org/llama.cpp/discussions/21961

Re: Qwen3.6-35B-A3B: Agentic coding power, now open to all

#424

Post by ThatPlayer »

aliljet wrote: post #73

I'm broadly curious how people are using these local models. Literally, how are they attaching harnesses to this and finding more value than just renting tokens from Anthropic of OpenAI?

I'm using the smaller vision models (Qwen3.5-4B currently) with Frigate, a FOSS self-hosted "AI" NVR. It's good enough at analyzing images to figure out mostly what's happening, and doesn't require the big knowledge base that bigger models have.

Also use a bigger model for summarizing or translating text, which I don't consume in realtime, so doesn't need to be fast. Would be a thing I could use OpenAI's batch APIs for if I did need something higher quality.

Re: Qwen3.6-35B-A3B: Agentic coding power, now open to all

#425

Post by quietsegfault »

simonw wrote: post #270

I've been running this on my laptop with the Unsloth 20.9GB GGUF in LM Studio: https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF/blob/mai... It drew a better pelican riding a bicycle than Opus 4.7 did! https://simonwillison.net/2026/Apr/16/qwen-beats-opus/

The qwen flamingo looks like it’s smoking’ a doobie.

Re: Qwen3.6-35B-A3B: Agentic coding power, now open to all

#426

Post by kanemcgrath »

kanemcgrath wrote: post #356

I have been using Qwen3.5-35B-A3B a lot in local testing, and it is by far the most capable model that could fit on my machine. I think quantization technology has really upped its game around these models, and there were two quants that blew me away Mudler APEX-I-Quality. then later I tried Byteshape Q3_K_S-3.40bpw Both made claims that seemed too good to be true, but I couldn't find any traces of lobotomization doi…

Now that I have tried out on a few tasks, Qwen3.6 is a huge jump in capability. It can make improvements to a project that qwen3.5 always struggled with.

Re: Qwen3.6-35B-A3B: Agentic coding power, now open to all

#427

Post by jadbox »

kanemcgrath wrote: post #356

I have been using Qwen3.5-35B-A3B a lot in local testing, and it is by far the most capable model that could fit on my machine. I think quantization technology has really upped its game around these models, and there were two quants that blew me away Mudler APEX-I-Quality. then later I tried Byteshape Q3_K_S-3.40bpw Both made claims that seemed too good to be true, but I couldn't find any traces of lobotomization doi…

Which one is best?

Re: Qwen3.6-35B-A3B: Agentic coding power, now open to all

#428

Post by rcxdude »

bigyabai wrote: post #186
amazingamazing wrote: post #8

More benchmaxxing I see. Too bad there’s no rig with 256gb unified ram for under $1000

taps the sign Unified Memory Is A Marketing Gimmeck. Industrial-Scale Inference Servers Do Not Use It.

Unified Memory is mainly how consumer hardware has enough RAM accessible by the GPU to run larger models, because otherwise the market segmentation jacks up the price substantially.

Re: Qwen3.6-35B-A3B: Agentic coding power, now open to all

#429

Post by Nav_Panel »

kylehotchkiss wrote: post #252
zozbot234 wrote: post #40

Earlier quoted context omitted.

This is just one model in the Qwen 3.6 series. They will most likely release the other small sizes (not much sense in keeping them proprietary) and perhaps their 122A10B size also, but the flagship 397A17B size seems to have been excluded.

How many people/hackernews can run a 397b param model at home? Probably like 20-30.

The point is that open weights turns puts inference on the open market, so if your model is actually good and providers want to serve it, it will drive costs down and inference speeds up. Like Cerebras running Qwen 3 235B Instruct at 1.4k tps for cheaper than Claude Haiku (let that tps number sink in for a second. For reference, Claude Opus runs ~30-40 tps, Claude Haiku at ~60. Several orders of magnitude difference). As a company developing models, it means you can't easily capture the inference margins even though I believe you get a small kickback from the providers.

So I understand why they wouldn't want to go open weight, but on the other hand, open weight wins you popularity/sentiment if the model is any good, researchers (both academic and other labs) working on your stuff, etc etc. Local-first usage is only part of the story here. My guess is Qwen 3.5 was successful enough that now they want to start reaping the profits. Unfortunately most of Qwen 3.5's success is because it's heavily (and successfully!) optimized for extremely long-context usage on heavily constrained VRAM (i.e. local) systems, as a result of its DeltaNet attention layers.

Re: Qwen3.6-35B-A3B: Agentic coding power, now open to all

#430

Post by nyrikki »

jchw wrote: post #259
dist-epoch wrote: post #222

Earlier quoted context omitted.

NVIDIA 5070 Ti can run Gemma 4 26B at 4-bit at 120 tk/s. Arc Pro B70 seems unexpectedely slow? Or are you using 8-bit/16-bit quants.

Unfortunately it really is running this slow with Llama.cpp, but of course that's with Vulkan mode. The VRAM capacity is definitely where it shines, rather than compute power. I am pretty sure that this isn't really optimal use of the cards, especially since I believe we should be able to get decent, if still sublinear, scaling with multiple cards. I am not really a machine learning expert, I'm curious to see if I ca…

Parallelism can be tricky and always has a cost, but don't discount the 3090 which is more expensive these days in that price bracket.

3090 llama.cpp (container in VM)

    unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_XL  105 t/s
    unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_XL  103 t/s
Still slow compaired to the

    ggml-org/gpt-oss-20b-GGUF 206 t/s
But on my 3x 1080 Ti 1x TITAN V getto machine I learned that multi gpu takes a lot of tuning no matter what. With the B70, where Vulkan has the CPU copy problem, and SYCL doesn't have a sponsor or enough volunteers, it will probably take a bit of profiling on your part.

There are a lot of variables, but PCIe bus speed doesn't matter that much for inference, but the internal memory bandwidth does, and you won't match that with PCIe ever.

To be clear, multicard Vulkan and absolutely SYCL have a lot of optimizations that could happen, but the only time two GPUs are really faster for inference is when one doesn't have enough ram to fit the entire model.

A 3090 has 936.2 GB/s of (low latency) internal bandwidth, while 16xPCIe5 only has 504.12, may have to be copied through the CPU, have locks, atomic operations etc...

For LLM inference, the bottleneck just usually going to be memory bandwidth which is why my 3090 is so close to the 5070ti above.

LLM next token prediction is just a form of autoregressive decoding and will primarily be memory bound.

As I haven't used the larger intel GPUs I can't comment on what still needs to be optimized, but just don't expect multiple GPUs to increase performance without some nvlink style RDMA support _unless_ your process is compute and not memory bound.

Post reply on HN