Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.8 27B

huggingface.co

Post reply on HN
Print view View original

801–810 of 848 posts

Re: Qwen 3.8 27B

#801

Post by HarHarVeryFunny »

Der_Einzige wrote: post #786
HarHarVeryFunny wrote: post #779

Earlier quoted context omitted.

> If you look at the hiring marketplace, being just marginally better than your peers can be very lucrative. I would say that in software this is completely false. Someone straight out of college, not very useful, makes 75-100K. Top level senior outside of FAANG is making twice that at best (and at least 10x more capable).

You told on yourself about being either European or from a flyover state.

I happen to be from NJ, but you're also not going to be making a Meta/Google salary in NY outside of NYC.

Re: Qwen 3.8 27B

#802

Post by brcmthrowaway »

c16 wrote: post #756

Big thank you to the Qwen team. 3.6 A3B was shocking good, and now I'm hoping they release an 3.8 A3B model too. Edit: Having used qwen3.8:27b-mlx on MBP M4 64GB, I get around ~45 tok/s. A3B would be great for smaller devices, but it's definitely usable. As I understand it it's a mixture of MLX and MTP.

Thats a huge tok/sec. Prompt prefill is the bottleneck

Re: Qwen 3.8 27B

#803

Post by ssvegeta »

xlayn wrote: post #466

The unsloth Q8kxl https://huggingface.co/unsloth/Qwen3.8-27B-GGUF for some reason is looping and going crazy on the think part (I tried to search for an email to let the guys know but didn't find one)... I used the bartowski one and that one doesn't have that issue https://huggingface.co/bartowski/Qwen3.8-27B-GGUF?show_file_... that's using llama.cpp llama-server \ -m ~/somePath/Qwen3.8-27B-UD-Q8_K_XL.gguf \ -np 1 --…

I'm seeing this infinite loop behavior during reasoning as well, for the exact same quant from Unsloth. Set presence penalty to 1 and the issue seemly went away, but having a penalty that high worries me for coding tasks. Will try the bartowski one now.

Re: Qwen 3.8 27B

#804

Post by jwr »

jwr wrote: post #127
TomGarden wrote: post #24

Any tips on the best approach at running this at an M4 Max 128GB? Token throughput was a bit slow with the last 27B one (MLX), ended up using the A3B variant but if I could get this one to reasonable speed I'd much prefer it.

I have an M4 Max (unfortunately 64GB). I have been running the Qwen 35B A3B one for a while now, after testing and benchmarking a number of models. That one was consistently the best in class for tasks like despamming, E-mail classification, OCR and dictation post-processing. It was also really fast (90 tokens/s). I'm benchmarking the 3.8 model now, it seems it is better (near-perfect score on my E-mail spam filterin…

FWIW, Qwen 3.8 27B is actually slower with MTP enabled, unless you generate a lot of code with it (and even then it's not obvious, because you'll likely spend most time thinking).

Re: Qwen 3.8 27B

#805

Post by searealist »

monster_truck wrote: post #774
CMay wrote: post #518

Credit where it's due. Qwen 3.8 27B is only the second local model after Gemma 4 that managed to correctly reason through one of my private benchmarks. It took 5x as many tokens to do it and 12m30s with MTP enabled, but it did do it. Gemma 4 reasoned through it more implicitly, while Qwen 3.8 reasoned more explicitly. Laguna and Muse Glimmer failed hard on it, though they're useful for other tasks. The VRAM usage see…

In my experience MTP's speed increase doesn't seem to justify the apparent loss of success at the edge, it would have to be at least 4x faster to meaningfully churn through the first 3 failures in the time it would have taken to do it once without What exactly are you doing that the prompt is eating an entire 65536 window? Surely it would be better to let it use any number of approaches that call tools to access that…

> In my experience MTP's speed increase doesn't seem to justify the apparent loss of success at the edge, it would have to be at least 4x faster to meaningfully churn through the first 3 failures in the time it would have taken to do it once without

Speaking in terms of wall clock, the expensive part of decode is fetching the weights from memory. Predicting and validating a bunch of tokens using the already fetched weights is insignificant in comparison. Even if you have a poor acceptance rate for predictions, you won't really see a slowdown vs not using MTP.

Re: Qwen 3.8 27B

#806

Post by prometheus1992 »

jedbrooke wrote: post #437
prometheus1992 wrote: post #402

Earlier quoted context omitted.

16GB mac mini - what chip? m4 pro i assume?

nope just my normal m2 mac mini. I bought it just as a normal computer to do my taxes and whatever, so it’s mind blowing that I can run this kind of AI workload on it. Well, “run” might be generous, it gets like 3-5tok/s I’m working on a setup that’s more geared towards running tasks overnight so the slow tok/s doesn’t matter as much

ahh gotcha. that speed sounds about right. i am running qwen 3.6 35bA3B(group-32 int4) on my m3 air (16gb) and get about 8 tokens per sec. i have not yet tried the 3.6 27B.

Re: Qwen 3.8 27B

#807

Post by nullc »

jamesblonde wrote: post #709

For the full weights, unoptimized on vLLM with 2 Nvidia 6000 RTX 48GBs connected by NVLink, i only get 14 tokens/sec with open-code. For batched operations, it climbs to 55 tokens/sec. For FP8, on a single Nvidia 6000 RTX 48GB, i get 13 tokens/sec on a single GPU and 46 tokens/sec batched.

On 2x RTX A6000 non-nvlink connected but communicating across the CPU, with llama cpp I get ~60 tok/s for Qwen3.8-27B-UD-Q8_K_XL without any batching.

Re: Qwen 3.8 27B

#808

Post by sgtwompwomp »

nobrains wrote: post #790
CMay wrote: post #518

Credit where it's due. Qwen 3.8 27B is only the second local model after Gemma 4 that managed to correctly reason through one of my private benchmarks. It took 5x as many tokens to do it and 12m30s with MTP enabled, but it did do it. Gemma 4 reasoned through it more implicitly, while Qwen 3.8 reasoned more explicitly. Laguna and Muse Glimmer failed hard on it, though they're useful for other tasks. The VRAM usage see…

"correctly reason through one of my private benchmarks" i would also like to make one myself for my testing. could you give a rough idea or an outline or point in the general direction on what to do?

You can try https://www.vals.ai/vals-smith for this, saw it recently.

Re: Qwen 3.8 27B

#809

Post by ElectricalUnion »

fwipsy wrote: post #671
lwansbrough wrote: post #515

Earlier quoted context omitted.

Perhaps this isn’t a new observation but the problem with LLMs is very clear with these. It’s a nice microcosm. The LLM will draw a fish companion (unprompted!) with a nice gradient but won’t get the pelican’s feet right. It’s obviously a problem of fundamental understanding and demonstrates that reasoning is more “directionless rigour”.

I'd like to see a human create a better pelican SVG without being able to look at the results. LLMs are language models, not (natively) vision models*. That's why it's a good benchmark: it engages LLM's logical reasoning in a way that we can check visually. The fact that they make errors which can be spotted visually, doesn't prove that LLMs are so far behind humans on language/logic tasks. Besides which, "fundamenta…

AFAIK the only well-known LLM with no "conventional" multimodal encoders required for audio and images is Gemma 4 12B.

Re: Qwen 3.8 27B

#810

Post by __s »

chr15m wrote: post #601
svdr wrote: post #562

Wow. This model is so good, and we have GLM 5.3 (seems great voor security related work) and Deepseek. In a few months we'll have Fable/Sol-like capabilities that are not coming from the big US companies. I feel as a programmer that that is more than enough. How wil OpenAI and Anthropic survive when frontier model intelligence becomes commoditized?

I stopped using Fable and Opus 5 because I literally can't understand the output. The waffle is so intense it no longer makes sense. I don't understand who is using those to get real work done. I'm using kimi-k3 on a real high level dev & analysis task at the moment (with Gemini flash for fast implementation) and it is fantastic. I think we have passed the point where frontier intelligence is commoditized.

Using it to get real work done:

* https://github.com/HarbourMasters/Shipwright/pull/7047

* https://github.com/HarbourMasters/Shipwright/pull/7070

just need to give it clear direction, guide it about how to properly use hooks / preserve decomp code, rewrite all the comments, go back & redo things when it spits out some ugly hack, ...

but this does save time, especially considering this is working with decomp code where many functions are random hex addresses

Post reply on HN