Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model

qwen.ai

Post reply on HN
Print view View original

261–270 of 482 posts

Re: Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model

#261

Post by finnjohnsen2 »

Since Gemma 4 came this easter the gap from self hosting models to Claude has decreased sigificantly I think. The gap is still huge it just that local models were extremely non-competitive before easter. So now it seems Qwen 3.6 is another bump up from Gemma 4 which is exciting if it is so. I keep an Opus close ofcourse, because these local models still wander off in the wrong direction and fails. Something Opus almost never does for me anymore.

But every time a local model gets me by - I feel closer to where I should be; writing code should still be free. Both free as in free beer, and free as in freedom.

My setup is a seperate dedicated Ubuntu machine with RTX 5090. Qwen 3.6:27b uses 29/32gb of vram when its working right this minute. I use Ollama in a non root podman instance. And I use OpenCode as ACP Service for my editor, which I highly recommend. ACP (Agent Client Protocol) is how the world should be in case you were asking, which you didnt :)

Exciting times and thank you Qwen team for making the world a better place in a world of Sam Altmans.

Re: Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model

#262

Post by fnordpiglet »

ozgrakkurt wrote: post #234
fnordpiglet wrote: post #84

Earlier quoted context omitted.

For coding often quality at the margin is crucial even at a premium. It’s not the same as cranking out spam emails or HN posts at scale. This is why the marginal difference between your median engineer and your P99 engineer is comp is substantial, while the marginal comp difference between your median pick and packer vs your P99 pick and packer isn’t. I’d also say it keeps the frontier shops competitive while costing…

I just assume any of those companies would steal my work and wouldn't care about it. Everything they have done so far indicates this. Running your own is the only option unless you really trust them or unless you have the option to sue them like some big companies can. Or if you don't really care then you can use the chineese one since it is cheaper. What makes you trust Anthropic more than Alibaba?

There’s a difference between stealing for model training and direct monitoring of actionable trade secrets and corporate espionage. Anthropic and OpenAI wouldn’t do this simply because they would be litigated out of existence and criminally investigated if they did. In China it’s an expected part of the corporate and legal structure with virtually no recourse for a foreign firm and when it’s in states interest domestic either. I’m surprised you don’t realize the US has fairly strong civil, criminal, and regulatory protections in place for theft of actionable material and reuse of corporate and trade secrets, let alone copyright materials. I assure you their ToS also do not allow them to do this and that in itself is a contractual obligation you can enforce and win in court.

Re: Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model

#263

Post by gdevenyi »

anonzzzies wrote: post #11

I wish that all announcements of models would show what (consumer) hardware you can run this on today, costs and tok/s.

You can point at the GGUF files and figure it out with your hardware here.

https://github.com/gdevenyi/huggingface-estimate

Re: Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model

#264

Post by hadlock »

anonym29 wrote: post #117
Aurornis wrote: post #23

Earlier quoted context omitted.

The 27B model they release directly would require significant hardware to run natively at 16-bit: A Mac or Strix Halo 128GB system, multiple high memory consumer GPUs, or an RTX 6000 workstation card. This is why they don’t advertise which consumer hardware it can run on: Their direct release that delivers these results cannot fit on your average consumer system. Most consumers don’t run the model they release direct…

You absolutely do not need to run at full BF16. The quality loss between BF16 (55.65 GB in GGUF) and Q8_0 (30.44 GB in GGUF) is essentially zero - think on the order of magnitude of +0.01-0.03 perplexity, or ~0.1-0.3% relative PPL increase. The quality loss between BF16 and Q4_K_M (18.66 GB in GGUF) is close to imperceptible, with perplexity changes in the +0.1-0.3 ballpark, or ~1-3% relative PPL increase. This would…

How many t/s output are you getting at Q4_K_M with 200k context on your Strix Halo if you ask it to add a new feature to a codebase.

Re: Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model

#265

Post by gdevenyi »

zargon wrote: post #219
bityard wrote: post #76

Earlier quoted context omitted.

There are infinite combinations of CPU/GPU capable of running LLMs locally. What most people do is buy the system they can afford and roughly meets their goals and then ball-park VRAM usage by looking at the model size and quantization. For more a detailed analysis, there are several online VRAM calculators. Here's one: https://smcleod.net/vram-estimator/ If you have a huggingface account, you can set your system con…

These calculators are almost entirely useless. They don't understand specific model architectures. Even the ones that try to support only specific models (like the apxml one) get it very wrong a lot of the time. For example, the one you linked, when I provide a Qwen3.5 27B Q_4_M GGUF [0], says that it will require 338 GB of memory with 16-bit kv cache. That is wrong by over an order of magnitude. [0] https://huggingf…

Mine does https://github.com/gdevenyi/huggingface-estimate

Re: Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model

#266

Post by manquer »

Abby_101 wrote: post #188
butz wrote: post #142

Are there any "optimized" models, that have lesser hardware requirements and are specialised in single programming language, e.g. C# ?

Sort of - there's Qwen3-Coder and the Codestral family, but those are still multi-language, just code-focused. For truly single-language specialization, the practical path is fine-tuning an existing base model on a narrow distribution rather than training from scratch. The issue with C# specifically is dataset availability. Open source C# code on GitHub is a fraction of Python/JS, and Microsoft hasn't released a publ…

Issues with C# not withstanding. It is not inherently bad idea for small models to trained on only specific languages like a JS/PY only model with declarative languages like HTML, CSS YAML, JSON, graph etc thrown in, probably could be more efficient for local use.

Fine-tuning / LoRA on basis the org code base would be make it more useful.

Re: Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model

#267

Post by proxysna »

PcChip wrote: post #133
proxysna wrote: post #31

Earlier quoted context omitted.

Qwen3.5-27B with a 4bit quant can be run on a 24G card with no problem. With 2 Nvidia L4 cards and some additional vllm flags, i am serving 10 developers at 20-25tok/sek, off-peak is around 40tok/sek. Developers are ok with that performance, but ofc they requested more GPU's for added throughput.

question: why not use something like Claude? is it for security reasons?

We do make Claude and Mistral available to our developers too. But, like you said, security. I, personally, do not understand how people in tech, put any amount of trust in businesses that are working in such a cutthroat and corrupt environment. But developers want to try new things and it is better to set up reasonable guardrails for when they want to use these thing by setting up a internal gateway and a set of reasonable policies.

And the other thing is that i want people to be able to experiment and get familiar with LLM's without being concerned about security, price or any other factor.

Re: Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model

#269

Post by jmspring »

Aurornis wrote: post #80
ryandrake wrote: post #67

Earlier quoted context omitted.

Yea, this is currently the confusing part of running local models for newbies: Even after you have decided which model you want to run, and which org's quantizations to use (let's just assume Unsloth's for example), there are often dozens of quantizations offered, and choosing among them is confusing. Say you have a GPU with 20GB of VRAM. You're probably going to be able to run all the 3-bit quantizations with no pro…

> Say you have a GPU with 20GB of VRAM. You're probably going to be able to run all the 3-bit quantizations with no problem, but which one do you choose? Unsloth offers[1] four of them: UD-IQ3_XXS, Q3_K_S, Q3_K_M, UD-Q3_K_XL There are actually two problems with this: First, the 3-bit quants are where the quality loss really becomes obvious. You can get it to run, but you’re not getting the quality you expected. The e…

qwen3.5 27b w/ 4bit quant works reasonably on a 3090.

Re: Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model

#270

Post by a_e_k »

zozbot234 wrote: post #126
anonym29 wrote: post #117

Earlier quoted context omitted.

You absolutely do not need to run at full BF16. The quality loss between BF16 (55.65 GB in GGUF) and Q8_0 (30.44 GB in GGUF) is essentially zero - think on the order of magnitude of +0.01-0.03 perplexity, or ~0.1-0.3% relative PPL increase. The quality loss between BF16 and Q4_K_M (18.66 GB in GGUF) is close to imperceptible, with perplexity changes in the +0.1-0.3 ballpark, or ~1-3% relative PPL increase. This would…

I'm all for running large MoE models on unified memory systems, but developers of inference engines should do a better job of figuring out how to run larger-than-total-RAM models on such systems, streaming in sparse weights from SSD but leveraging the large unified memory as cache. This is easily supported with pure-CPU inference via mmap, but there is no obvious equivalent when using the GPU for inference.

At least for the CPU/GPU split, llama.cpp recently added a `--fit` parameter (might default to on now?) that pairs with a `--fitc CONTEXTSIZE` parameter. That new feature will automatically look at your available VRAM and try to figure out a good CPU/GPU split for large models that leaves enough room for the context size that you request.
Post reply on HN