Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.8 27B

huggingface.co

Post reply on HN
Print view View original

371–380 of 848 posts

Re: Qwen 3.8 27B

#371

Post by rs38 »

jacquesm wrote: post #340
hypfer wrote: post #109

Since it might be helpful to some, here's my current commandline for llama.cpp running on an RTX 4090 with my monitor moved to the iGPU to free up all of its VRAM. llama-server -m Qwen3.8-27B-IQ4_NL.gguf --mmproj mmproj-BF16.gguf -c 170000 --parallel 1 -ngl -1 --cache-type-k q8_0 --cache-type-v q8_0 -b 1024 -ub 512 --flash-attn on --no-context-shift --no-mmproj-offload --spec-type draft-mtp --spec-draft-n-max 5 --spe…

I absolutely love this comment. I wished there was a website where people would post their working command lines as well as what hardware they are using to run that stuff on + tokens / sec prefill + gen.

pretty sure this exists already...

Re: Qwen 3.8 27B

#372

Post by g023 »

All this performance at such small model sizes, why are the API fees so high for the AI monopolists on this side of the world?

Re: Qwen 3.8 27B

#373

Post by mrbonner »

hypfer wrote: post #109

Since it might be helpful to some, here's my current commandline for llama.cpp running on an RTX 4090 with my monitor moved to the iGPU to free up all of its VRAM. llama-server -m Qwen3.8-27B-IQ4_NL.gguf --mmproj mmproj-BF16.gguf -c 170000 --parallel 1 -ngl -1 --cache-type-k q8_0 --cache-type-v q8_0 -b 1024 -ub 512 --flash-attn on --no-context-shift --no-mmproj-offload --spec-type draft-mtp --spec-draft-n-max 5 --spe…

Jeez, llama.c++ is becoming the ffmpeg cargo cult CLI now

Re: Qwen 3.8 27B

#374

Post by beembeem »

jacquesm wrote: post #340
hypfer wrote: post #109

Since it might be helpful to some, here's my current commandline for llama.cpp running on an RTX 4090 with my monitor moved to the iGPU to free up all of its VRAM. llama-server -m Qwen3.8-27B-IQ4_NL.gguf --mmproj mmproj-BF16.gguf -c 170000 --parallel 1 -ngl -1 --cache-type-k q8_0 --cache-type-v q8_0 -b 1024 -ub 512 --flash-attn on --no-context-shift --no-mmproj-offload --spec-type draft-mtp --spec-draft-n-max 5 --spe…

I absolutely love this comment. I wished there was a website where people would post their working command lines as well as what hardware they are using to run that stuff on + tokens / sec prefill + gen.

What stackoverflow should have become.

Re: Qwen 3.8 27B

#375

Post by wgd »

dannyw wrote: post #234
Casteil wrote: post #219

Earlier quoted context omitted.

Yeah, that's probably the answer given that it apparently defaults to 'xhigh'.

Probably helps it score a little bit better in benchmarks :) `medium` seems like a nice balance so far; along with some light steering to vary think effort as needed for task and being pragmatic.

Interestingly 'medium' is the closest thing the _model itself_ has to a default thinking level. The chat template injects directions [1] at the very start of the system message when the reasoning effort is 'xhigh' or 'low' but 'medium' implicitly just means no added reasoning-level instructions.

[1] The specific directions are "Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer." and "Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration."

Re: Qwen 3.8 27B

#376

Post by app13 »

XCSme wrote: post #348
swalsh wrote: post #338

WOW, my first try running on my 2 3090's, it was a bit slow... but it FEELS like opus 4.5, i gave it an image and a broad overview of what I wanted it to build, and it built the whole thing from beginning to end.

Why slow? I see ~50tps on a single 3090

Whats your setup? I have a single 3090 and am struggling to get it purring

Re: Qwen 3.8 27B

#377

Post by SwellJoe »

KronisLV wrote: post #10

I hope really badly that we'll get a new 35B A3B or similar MoE model! I also miss the Qwen 3 Coder Next, which was 80B A3B, there are quite a few use cases where a non-dense model Also alternate link for viewing the images without signing in: https://xcancel.com/Alibaba_Qwen/status/2088280182356611304

AgentWorld is a pretty recent MoE release from the Qwen folks with quite a bit better performance than 3.6, released around the same time as 3.7 Plus/Max, I guess. Not sure why it didn't get more attention, as it is definitely better than 3.6 35B A3B on all dimensions, but especially for agentic use. Still nowhere near the dense models (even 3.6 27B), but clearly an upgrade in the small Qwen MoE line.

https://huggingface.co/Qwen/Qwen-AgentWorld-35B-A3B

KAT Coder is another Qwen 3.6 MoE fine-tune that also improves on 3.6 by a measurable amount.

But, I do hope for a bigger MoE, in the 70B to 120B range, something in the Coder Next lineage. I've got a Strix Halo that isn't getting used to its best ability because the best models all run fine on my faster desktop dual 32GB GPU setup, and they run too slow for comfort on the Strix Halo.

Re: Qwen 3.8 27B

#378

Post by vel0city »

cjbprime wrote: post #282
hypfer wrote: post #109

Since it might be helpful to some, here's my current commandline for llama.cpp running on an RTX 4090 with my monitor moved to the iGPU to free up all of its VRAM. llama-server -m Qwen3.8-27B-IQ4_NL.gguf --mmproj mmproj-BF16.gguf -c 170000 --parallel 1 -ngl -1 --cache-type-k q8_0 --cache-type-v q8_0 -b 1024 -ub 512 --flash-attn on --no-context-shift --no-mmproj-offload --spec-type draft-mtp --spec-draft-n-max 5 --spe…

Hm, I have a 4090 as well, and: $ build/bin/llama-server -m Qwen3.8-27B-IQ4_NL.gguf --mmproj mmproj-BF16.gguf -c 170000 --parallel 1 -ngl -1 --cache-type-k q8_0 --cache-type-v q8_0 -b 1024 -ub 512 --flash-attn on --no-context-shift --no-mmproj-offload --spec-type draft-mtp --spec-draft-n-max 5 --spec-default --cache-type-k-draft q4_0 --cache-type-v-draft q4_0 --threads 24 --jinja --reasoning on -fit off 0.02.993.689…

>But nvidia-smi only showed Xorg using 200M out of the 24G, so why would a 911M alloc fail?

That was just the last buffer allocation request that failed, it didn't tell you by how much it failed by. It could have failed it by a few kilobytes, it could have failed it by 910MB. One would guess it probably failed it by a couple hundred megabytes in the end judging by your results.

Re: Qwen 3.8 27B

#379

Post by gip »

Balinares wrote: post #248

I wonder if Anthropic and OpenAI possibly missed the window to go public. A 27B open-weight model trading blows with the SOTA from just half a year ago is not great news for trillion-dollar investments...

The real target for Anthropic, OpenAI and tons of others are large enterprises and workflows. I think they will do all right if they execute well.

Re: Qwen 3.8 27B

#380

Post by bilekas »

formerly_proven wrote: post #260
bilekas wrote: post #182

Earlier quoted context omitted.

You sound like your trying to reassure yourself of something. I sure hope my boss doesn't think he built my work! He'd probably get fired pretty quickly during on call!

> I sure hope my boss doesn't think he built my work! Most managers do though?

I've worked with a good few types of managers of the years, none would take ownership of my work. The end result? Sure, for shareholders/this managers that is fine.

But that's not the same as me being needed to be on a call with integration teams and having to run the call. It's just a layer of abstraction for management. I own the creation side of the product.

Who created it is still me.

Post reply on HN