Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.8 27B

huggingface.co

Post reply on HN
Print view View original

221–230 of 848 posts

Re: Qwen 3.8 27B

#221

Post by usagisushi »

D4Ha wrote: post #133
hypfer wrote: post #109

Since it might be helpful to some, here's my current commandline for llama.cpp running on an RTX 4090 with my monitor moved to the iGPU to free up all of its VRAM. llama-server -m Qwen3.8-27B-IQ4_NL.gguf --mmproj mmproj-BF16.gguf -c 170000 --parallel 1 -ngl -1 --cache-type-k q8_0 --cache-type-v q8_0 -b 1024 -ub 512 --flash-attn on --no-context-shift --no-mmproj-offload --spec-type draft-mtp --spec-draft-n-max 5 --spe…

Do you find it useful or worthwhile to split a large LLM across two GPUs on a desktop? If you've tried it, what worked well and what didn't? I'm especially interested in mismatched VRAM setups, e.g. a 16 GB GPU + a 24 GB GPU. How much overhead did you see from inter-GPU transfers, and did the extra usable VRAM outweigh the performance hit?

A hetero-GPU setup is definitely cost-effective if you don't strictly require the raw speed of a top-tier card like 5090. Just keep in mind that the total throughput will also be bottlenecked by the slower card.

To provide some anecdotal data, here is how my 5090 + 3060 setup performs with Qwen 3.8 27B (Unsloth's UD-Q4 with MTP):

  Single 5090: 101 t/s (TG), 2650 t/s (PP)
  5090 + 3060: 53 t/s (TG), 1700 t/s (PP)
For reference, here are also some numbers from my 4060ti + 3060 (16GB + 12GB) setup. [0]

[0]: https://news.ycombinator.com/item?id=48700091

Re: Qwen 3.8 27B

#222

Post by ThreatSystems »

bmitc wrote: post #167
skrebbel wrote: post #159

Earlier quoted context omitted.

> LM Studio doesn't work behind proxies. Woa, is that still a thing? You mean like SOCKS5 stuff that you have to manually configure in every application that uses the internet? I mean maybe I'm just living under a rock but I feel like that's a rather niche situation you got there.

> I feel like that's a rather niche situation you got there Every big company in the world uses a network proxy. LM Studio, as far as I can tell, cannot be configured to work behind such proxies.

If you're on Linux you can probably use proxychains.

Re: Qwen 3.8 27B

#223

Post by awkwardpotato »

literoldolphin wrote: post #211

Why is anyone even using video cards these days? You may as well be burning cash. This is the perfect candidate for just splattering it on your nvme and then reading it off there and into memory. All of these run perfectly fine on simple m4 silicone: https://github.com/drumih/turbo-fieldfare https://github.com/leonickson1/Swiftlet https://github.com/sqliteai/warp

Those are all for MoE models. And I prefer measuring my tokens in t/s instead of s/t

Re: Qwen 3.8 27B

#224

Post by aappleby »

Almondsetat wrote: post #132

The $1500 Intel B70 with 32GB of VRAM can run this model at max context with good performance, btw. If you don't want to drop $5-10k for running DeepSeek this is your best budget option for local refactor/small scale dev help

I have a B70, what llama options are you using and what performance are you seeing?

Re: Qwen 3.8 27B

#225

Post by monkmartinez »

Qwen3.6-27B has been the main LLM powering my little agentic stack. I have adopted the test and verify approach to any models allowed to run on my machine. When the "heretic" version drops, I will fire up the harness and test. Super excited to see how it stacks up against Qwen3.6!!!

Re: Qwen 3.8 27B

#226

Post by IronWolve »

satvikpendem wrote: post #179
Casteil wrote: post #82

One thing a lot of people don't seem to factor when hyping Qwen is how much models like this tend to 'overthink' with seemingly endless 'second guessing'. 3.8 seems no different from what I've tried thus far. As capable as it is, it's hard to justify using it when a competing model (e.g. Gemma4:26b-a3b) can consistently achieve the same or similar response with only 1/10th as many 'thinking' tokens, achieve much high…

Reduce or turn off thinking: https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates

Thank you, this is exactly what I needed.

Re: Qwen 3.8 27B

#227

Post by Aurornis »

bmitc wrote: post #167
skrebbel wrote: post #159

Earlier quoted context omitted.

> LM Studio doesn't work behind proxies. Woa, is that still a thing? You mean like SOCKS5 stuff that you have to manually configure in every application that uses the internet? I mean maybe I'm just living under a rock but I feel like that's a rather niche situation you got there.

> I feel like that's a rather niche situation you got there Every big company in the world uses a network proxy. LM Studio, as far as I can tell, cannot be configured to work behind such proxies.

> Every big company in the world uses a network proxy.

It's becoming more rare, now.

A lot of the universal truths about corporate networks from the early 2000s are no longer true today. Some companies are stuck in their ways though.

The overlap between companies that require someone to use a network proxy and companies that have GPU-equipped machines with enough RAM for LLMs and and that allow people to download and run executables of their choosing has to be small.

Re: Qwen 3.8 27B

#228

Post by regularfry »

davidw wrote: post #187

I don't know much about the production of these models. How hard would it be to 'fork' something like this and have it not be full of CCP indoctrination?

Look for `heretic` fine-tunes in the next couple of days.

Re: Qwen 3.8 27B

#229

Post by smallmancontrov »

effdee wrote: post #208
nubg wrote: post #165

Earlier quoted context omitted.

just to clarify. yes YOU built it. just because you used some tool doesn't mean the idea, prompting, reprompting, babysitting was not your creative input and effort. put differently, if you put a random person infront of whatever model you used (say, a 50yo receptionist at a pharmacy in india), they would not have been able to create that, because they would have lacked the motivation, idea, background knowledge, tas…

Some people will now argue it was the chisel—not Michelangelo—who created David.

"Carve me a naked guy. Make no mistakes."

Re: Qwen 3.8 27B

#230

Post by b112 »

effdee wrote: post #208
nubg wrote: post #165

Earlier quoted context omitted.

just to clarify. yes YOU built it. just because you used some tool doesn't mean the idea, prompting, reprompting, babysitting was not your creative input and effort. put differently, if you put a random person infront of whatever model you used (say, a 50yo receptionist at a pharmacy in india), they would not have been able to create that, because they would have lacked the motivation, idea, background knowledge, tas…

Some people will now argue it was the chisel—not Michelangelo—who created David.

No, it's the difference between management and direct work.

None would claim they chiseled anything, if it was 3D printed. They may claim they designed something.

Post reply on HN