Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.8 27B

huggingface.co

Post reply on HN
Print view View original

391–400 of 848 posts

Re: Qwen 3.8 27B

#391

Post by amluto »

augusto-moura wrote: post #368
Balinares wrote: post #248

I wonder if Anthropic and OpenAI possibly missed the window to go public. A 27B open-weight model trading blows with the SOTA from just half a year ago is not great news for trillion-dollar investments...

IMO AI companies are selling the infrastructure and easier access to the models, beyond competing to get the smarter model. We can run Qwen locally, but at what cost? A good enough GPU for running 27B is more than 4,000 USD where I live (Brazil). Even if I were to spend 100s of hours in credits I would never reach a point where the 4000 USD investment pays off. Unless I'm doing it for the sport, building my own LLM r…

> RTX 3090 for each and every employee

That’s quite inefficient. Get 2x RTX 6000 Pro and run DeepSeek v4 Flash for the whole company. This may or may not be cost effective compared to cheap third party providers, but it lets you own your own data, is not actually that expensive, and reports suggest that you can get 2k tok/s or so from a setup like this (at respectable batch size, but that’s exactly what you want in this situation).

I’m not sure I believe that the unquantized performance is as good as people are claiming, but even light quantization would make it fit easily.

Re: Qwen 3.8 27B

#392

Post by hypfer »

Aurornis wrote: post #157
hypfer wrote: post #109

Since it might be helpful to some, here's my current commandline for llama.cpp running on an RTX 4090 with my monitor moved to the iGPU to free up all of its VRAM. llama-server -m Qwen3.8-27B-IQ4_NL.gguf --mmproj mmproj-BF16.gguf -c 170000 --parallel 1 -ngl -1 --cache-type-k q8_0 --cache-type-v q8_0 -b 1024 -ub 512 --flash-attn on --no-context-shift --no-mmproj-offload --spec-type draft-mtp --spec-draft-n-max 5 --spe…

> --cache-type-k q8_0 --cache-type-v q8_0 In my tests, even Q8 quantization for the KV cache comes with notable drops in performance for longer tasks. It does provide more context length in limited RAM budgets, but the longer context tasks are where KV quantization starts to show problems. It’s basically unnoticeable for simple and short tasks. > --spec-draft-n-max 5 5 is a lot of tokens to draft. Are you really seei…

Okay, so, I did some actual coding with qwen3.8 in the last 2 hours, and if I (let the agent) take the average of all those "mean len" loglines, and let them get weighted by how much happened leading up to that logline, it's an average of 3.33

The unweighted average is 4.1

So with qwen3.8 as well, I think I would not go below 3 in this setup with this workload. Meaning 4, and, to push it a bit further, 5.

Also, cool that it carries over from qwen3.6 like that.

Re: Qwen 3.8 27B

#393

Post by alfiedotwtf »

usagisushi wrote: post #221
D4Ha wrote: post #133

Earlier quoted context omitted.

Do you find it useful or worthwhile to split a large LLM across two GPUs on a desktop? If you've tried it, what worked well and what didn't? I'm especially interested in mismatched VRAM setups, e.g. a 16 GB GPU + a 24 GB GPU. How much overhead did you see from inter-GPU transfers, and did the extra usable VRAM outweigh the performance hit?

A hetero-GPU setup is definitely cost-effective if you don't strictly require the raw speed of a top-tier card like 5090. Just keep in mind that the total throughput will also be bottlenecked by the slower card. To provide some anecdotal data, here is how my 5090 + 3060 setup performs with Qwen 3.8 27B (Unsloth's UD-Q4 with MTP): Single 5090: 101 t/s (TG), 2650 t/s (PP) 5090 + 3060: 53 t/s (TG), 1700 t/s (PP) For ref…

I wonder if turning on GGML_CUDA_NCCL would help reduce delay between cards

Re: Qwen 3.8 27B

#394

Post by me_bx »

satvikpendem wrote: post #179
Casteil wrote: post #82

One thing a lot of people don't seem to factor when hyping Qwen is how much models like this tend to 'overthink' with seemingly endless 'second guessing'. 3.8 seems no different from what I've tried thus far. As capable as it is, it's hard to justify using it when a competing model (e.g. Gemma4:26b-a3b) can consistently achieve the same or similar response with only 1/10th as many 'thinking' tokens, achieve much high…

Reduce or turn off thinking: https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates

In my experience with Qwen3.6 35B-A3B, disabling thinking made the model generate inaccurate replies. Ask it for the recipe of egg salad and it gives you the recipe of an omelette.

Did I miss something, is it possible to have that model be reliable without thinking?

Re: Qwen 3.8 27B

#395

Post by hypfer »

mrbonner wrote: post #373
hypfer wrote: post #109

Since it might be helpful to some, here's my current commandline for llama.cpp running on an RTX 4090 with my monitor moved to the iGPU to free up all of its VRAM. llama-server -m Qwen3.8-27B-IQ4_NL.gguf --mmproj mmproj-BF16.gguf -c 170000 --parallel 1 -ngl -1 --cache-type-k q8_0 --cache-type-v q8_0 -b 1024 -ub 512 --flash-attn on --no-context-shift --no-mmproj-offload --spec-type draft-mtp --spec-draft-n-max 5 --spe…

Jeez, llama.c++ is becoming the ffmpeg cargo cult CLI now

-funroll-loops

Re: Qwen 3.8 27B

#396

Post by qiine »

hypfer wrote: post #109

Since it might be helpful to some, here's my current commandline for llama.cpp running on an RTX 4090 with my monitor moved to the iGPU to free up all of its VRAM. llama-server -m Qwen3.8-27B-IQ4_NL.gguf --mmproj mmproj-BF16.gguf -c 170000 --parallel 1 -ngl -1 --cache-type-k q8_0 --cache-type-v q8_0 -b 1024 -ub 512 --flash-attn on --no-context-shift --no-mmproj-offload --spec-type draft-mtp --spec-draft-n-max 5 --spe…

what --no-context-shift is for ?

Re: Qwen 3.8 27B

#397

Post by qeternity »

peri-cl wrote: post #322
peri-cl wrote: post #27

Earlier quoted context omitted.

Same here! Qwen3.6-35B-A3B is the only local model I've found that runs reasonably on my iGPU. Looks like me and and my noisily-wheezing laptop will be sitting out this upgrade.

[self-reply because comment edit window ended]: I now have a head-to-head benchmark. On my wheezy laptop (specs in sibling comment), this new dense model, Qwen 3.8 27B, gets ~4 tokens/second on generation. The older mixture-of-experts model, Qwen 3.6-35B-A3B, gets ~20 tokens/s. MoE is literally 5x times faster (on CPU) than comparable dense Qwen! Tested: Qwen3.6-35B-A3B-UD-Q4_K_XL Qwen3.8-27B-UD-Q4_K_XL

35A3 might be more comparable to 10 dense.

27 dense is far more capable than 35A3.

Re: Qwen 3.8 27B

#398

Post by kzrdude »

isomorphic wrote: post #367
Casteil wrote: post #26

Earlier quoted context omitted.

I'm hoping too that they'll put out some MoE variants. Qwen3.5:122b:a10b can run about twice as fast as this 27b dense model. Edit: Like its predecessors, 3.8 seems really inclined to overthinking, and on a 27b dense model that's kind of painful. I think I'm going to stick with gemma4:26b-a3b as my go-to because it runs about 4x as fast and tends to only need a fraction of the tokens in its 'thinking' stage to get th…

"inclined to overthinking," holy cats you're not kidding! On a Mac mini M4 Pro 64GB I prompted it with "svg owl" and it thought for 17m12s, outputting 36.3KiB of thinking chatter. It did end up producing a 20.2KiB HTML+JS+SVG file with a very nice owl, including cursor-tracking animation, but it ran for more than a half hour! The MoE models are stupendously faster.

I think that's called test-time scaling i.e using more tokens at infer time to squeeze out higher model performance. That's must be part of the explanation for good benchmark results.

Re: Qwen 3.8 27B

#399

Post by yalok »

natch wrote: post #334

Apart from model performance, what harness are people using to come close to Claude Code or Codex workflow styles with tool use, conversations, loops, remote control, etc.?

and more specifically - what harness is known to be the best fit for Qwen local models, and are there any evals/benchmarks for harness+model pairs?

Re: Qwen 3.8 27B

#400

Post by kzrdude »

qeternity wrote: post #397
peri-cl wrote: post #322

Earlier quoted context omitted.

[self-reply because comment edit window ended]: I now have a head-to-head benchmark. On my wheezy laptop (specs in sibling comment), this new dense model, Qwen 3.8 27B, gets ~4 tokens/second on generation. The older mixture-of-experts model, Qwen 3.6-35B-A3B, gets ~20 tokens/s. MoE is literally 5x times faster (on CPU) than comparable dense Qwen! Tested: Qwen3.6-35B-A3B-UD-Q4_K_XL Qwen3.8-27B-UD-Q4_K_XL

35A3 might be more comparable to 10 dense. 27 dense is far more capable than 35A3.

So you happen to know how to demonstrate this higher capacity, what do we mean by that? Can understand bigger codebases? And how do we explain it, what part of the model does it?
Post reply on HN