Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.8 27B

huggingface.co

Post reply on HN
Print view View original

131–140 of 848 posts

Re: Qwen 3.8 27B

#131

Post by Phemist »

Casteil wrote: post #26
KronisLV wrote: post #10

I hope really badly that we'll get a new 35B A3B or similar MoE model! I also miss the Qwen 3 Coder Next, which was 80B A3B, there are quite a few use cases where a non-dense model Also alternate link for viewing the images without signing in: https://xcancel.com/Alibaba_Qwen/status/2088280182356611304

I'm hoping too that they'll put out some MoE variants. Qwen3.5:122b:a10b can run about twice as fast as this 27b dense model. Edit: Like its predecessors, 3.8 seems really inclined to overthinking, and on a 27b dense model that's kind of painful. I think I'm going to stick with gemma4:26b-a3b as my go-to because it runs about 4x as fast and tends to only need a fraction of the tokens in its 'thinking' stage to get th…

Did you try the claude reasoning traces finetune for qwen3.6? I find that it works muuch better. I assume the same 3.8 finetune will be released at some pointas well.

Edit: link - https://huggingface.co/rico03/Qwen3.6-27B-Claude-Opus-Reason...

Re: Qwen 3.8 27B

#132

Post by Almondsetat »

The $1500 Intel B70 with 32GB of VRAM can run this model at max context with good performance, btw. If you don't want to drop $5-10k for running DeepSeek this is your best budget option for local refactor/small scale dev help

Re: Qwen 3.8 27B

#133

Post by D4Ha »

hypfer wrote: post #109

Since it might be helpful to some, here's my current commandline for llama.cpp running on an RTX 4090 with my monitor moved to the iGPU to free up all of its VRAM. llama-server -m Qwen3.8-27B-IQ4_NL.gguf --mmproj mmproj-BF16.gguf -c 170000 --parallel 1 -ngl -1 --cache-type-k q8_0 --cache-type-v q8_0 -b 1024 -ub 512 --flash-attn on --no-context-shift --no-mmproj-offload --spec-type draft-mtp --spec-draft-n-max 5 --spe…

Do you find it useful or worthwhile to split a large LLM across two GPUs on a desktop?

If you've tried it, what worked well and what didn't? I'm especially interested in mismatched VRAM setups, e.g. a 16 GB GPU + a 24 GB GPU.

How much overhead did you see from inter-GPU transfers, and did the extra usable VRAM outweigh the performance hit?

Re: Qwen 3.8 27B

#134

Post by minimaltom »

xlayn wrote: post #42

The file "Just loads" on llama.cpp, the Unsloth https://huggingface.co/unsloth/Qwen3.8-27B-GGUF is an MTP file, I see mostly the same speed on pp and generation. There has to be something wrong with those benchmarks, I find extremely hard to believe a 27B model can work similar or exceed opus 4.6.

Worth distinguishing knowledge/task benchmarks from IF / agentic. It doesn't seem out of the question that you can have a small model thats generally good at instruction following and long-horizon agentic, as usually in those cases any requisite knowledge is in the context.

Most of the benchmark improvements afaict are in agentic and instruction following benchmarks.

Re: Qwen 3.8 27B

#135

Post by arjie »

ericd wrote: post #105
arjie wrote: post #96

I use the Qwens as a vision model for my DeepSeek V4 Flashes to handle. But the Qwens run on old RTX A6000 Ampere. Does anyone know if there's any news about INT4/AWQ quants for the RTX A6000?

Was recently thinking about doing something similar, do you basically just have the qwens describe what they see for the flashes? Was considering adding a LoRa/vision head to Flash, but seems like it could take a while to get it right. If DSv4 Flash was multimodal, I’d probably be done model shopping for a while

Same, with a multimodal DSv4 Flash I would just stop paying attention to things. Very smart, and at 260 tok/s it's too fast to care about anything else. If you ever graft something like that I would love to hear about it.

Yes, I have a very dumb flow. The harness has a describe_image tool that takes an image and a prompt and so DSv4 Flash uses it to get an idea of what it's looking at.

Re: Qwen 3.8 27B

#136

Post by rhdunn »

bewareofscams wrote: post #54
KronisLV wrote: post #45

Earlier quoted context omitted.

> ...but no. They do not beat opus on real-world usage. I agree, but then we just need meaningful benchmarks that clearly show that! Otherwise it's hand waving about something that should be put on paper in quantifiable terms.

Only useful benchmarks are those you (in particular) don't have access to.

The only useful benchmarks are those you've created for your specific workflow. Only then can you assess whether a given model is better or worse for what you are using it for.

There are tools like promptfoo designed for this.

Re: Qwen 3.8 27B

#137

Post by Alifatisk »

kennywinker wrote: post #107
Alifatisk wrote: post #59

Earlier quoted context omitted.

> I'd gladly take A5B or A8B or even A10B as a sort of middle ground. Whats up with focusing on the active param count? Do yall fiddle with the weights or something?

Total param count decides how much vram you need to run it. Active param count decides how fast it runs. My 10 year old GPU can load quantized 35B or 27B, but it can’t process 27B parameters per token faster than 2-4tok/s, while it can do A3B at >40tok/s

Thank you Kenny

Re: Qwen 3.8 27B

#138

Post by bmitc »

hypfer wrote: post #109

Since it might be helpful to some, here's my current commandline for llama.cpp running on an RTX 4090 with my monitor moved to the iGPU to free up all of its VRAM. llama-server -m Qwen3.8-27B-IQ4_NL.gguf --mmproj mmproj-BF16.gguf -c 170000 --parallel 1 -ngl -1 --cache-type-k q8_0 --cache-type-v q8_0 -b 1024 -ub 512 --flash-attn on --no-context-shift --no-mmproj-offload --spec-type draft-mtp --spec-draft-n-max 5 --spe…

Lol at that command. Why is this stuff so hard to run locally? I've spent a few days trying to figure it all out and haven't been able to. LM Studio doesn't work behind proxies. Ollama is confusing and doesn't seem to support Qwen3? And Llama.cpp is your command.

I just want to run ` ` with some default parameters set and for it to run locally.

Re: Qwen 3.8 27B

#139

Post by bogzz »

Almondsetat wrote: post #132

The $1500 Intel B70 with 32GB of VRAM can run this model at max context with good performance, btw. If you don't want to drop $5-10k for running DeepSeek this is your best budget option for local refactor/small scale dev help

Oh, can it work with the /v1/completions/ auto-complete endpoint?

Re: Qwen 3.8 27B

#140

Post by [deleted] »

D4Ha wrote: post #133
hypfer wrote: post #109

Since it might be helpful to some, here's my current commandline for llama.cpp running on an RTX 4090 with my monitor moved to the iGPU to free up all of its VRAM. llama-server -m Qwen3.8-27B-IQ4_NL.gguf --mmproj mmproj-BF16.gguf -c 170000 --parallel 1 -ngl -1 --cache-type-k q8_0 --cache-type-v q8_0 -b 1024 -ub 512 --flash-attn on --no-context-shift --no-mmproj-offload --spec-type draft-mtp --spec-draft-n-max 5 --spe…

Do you find it useful or worthwhile to split a large LLM across two GPUs on a desktop? If you've tried it, what worked well and what didn't? I'm especially interested in mismatched VRAM setups, e.g. a 16 GB GPU + a 24 GB GPU. How much overhead did you see from inter-GPU transfers, and did the extra usable VRAM outweigh the performance hit?

[deleted]
Post reply on HN