Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen3-Next

qwen.ai

Post reply on HN
Print view View original

221–230 of 240 posts

Re: Qwen3-Next

#221

Post by furyofantares »

EnPissant wrote: post #218
furyofantares wrote: post #216

Earlier quoted context omitted.

Do you know if it's doing what was described earlier, when I run it with all layers on GPU - paging an expert in every time the expert changes? Each expert is only 5.1B parameters.

It makes absolutely no sense to do what OP described. The decode stage is bottlenecked on memory bandwidth. Once you pull the weights from system RAM, your work is almost done. To then gigabytes of weights PER TOKEN over PCIE to do some trivial computation on the GPU is crazy. What actually happens is you run some or all of the MoE layers on the CPU from system RAM. This can be tolerable for smaller MoE models, but k…

I tried a few things and checked CPU usage in Task Manager to see how much work the CPU is doing.

KV Cache in GPU and 36/36 layers in GPU: CPU usage under 3%.

KV Cache in GPU and 35/36 layers in GPU: CPU usage at 35%.

KV Cache moved to CPU and 36/36 layers in GPU: CPU usage at 34%.

I believe you that it doesn't make sense to do it this way, it is slower, but it doesn't appear to be doing much of anything on the CPU.

You say gigabytes of weights PER TOKEN, is that true? I think an expert is about 2 GB, so a new expert is 2 GB, sure - but I might have all the experts for the token already in memory, no?

Re: Qwen3-Next

#222

Post by furyofantares »

furyofantares wrote: post #216
EnPissant wrote: post #215

Earlier quoted context omitted.

FWIW, that's a 80GB model and you also need kv cache. You'd need 96GBish to run on the GPU.

Do you know if it's doing what was described earlier, when I run it with all layers on GPU - paging an expert in every time the expert changes? Each expert is only 5.1B parameters.

^ Er, misspoke, each expert is at most .9 B parameters there's 128 experts. 5.1 B is number of active parameters (4 experts + some other parameters).

Re: Qwen3-Next

#223

Post by EnPissant »

furyofantares wrote: post #221
EnPissant wrote: post #218

Earlier quoted context omitted.

It makes absolutely no sense to do what OP described. The decode stage is bottlenecked on memory bandwidth. Once you pull the weights from system RAM, your work is almost done. To then gigabytes of weights PER TOKEN over PCIE to do some trivial computation on the GPU is crazy. What actually happens is you run some or all of the MoE layers on the CPU from system RAM. This can be tolerable for smaller MoE models, but k…

I tried a few things and checked CPU usage in Task Manager to see how much work the CPU is doing. KV Cache in GPU and 36/36 layers in GPU: CPU usage under 3%. KV Cache in GPU and 35/36 layers in GPU: CPU usage at 35%. KV Cache moved to CPU and 36/36 layers in GPU: CPU usage at 34%. I believe you that it doesn't make sense to do it this way, it is slower, but it doesn't appear to be doing much of anything on the CPU.…

gpt-oss-120b chooses 4 experts per token and combines them.

I don't know how lmstudio works. I only know the fundamentals. There is not way it's sending experts to the GPU per token. Also, the CPU doesn't have much work to do. It's mostly waiting on memory.

Re: Qwen3-Next

#224

Post by DiabloD3 »

EnPissant wrote: post #184
DiabloD3 wrote: post #56

Earlier quoted context omitted.

I don't load all the MoE layers onto my GPU, and I have only about a 15% reduction in token generation speed while maintaining a model 2-3 times larger than VRAM alone.

The slowdown is far more than 15% for token generation. Token generation is mostly bottlenecked by memory bandwidth. Dual channel DDR5-6000 has 96GB/s and A rtx 5090 has 1.8TB/s. See my other comment when I show 5x slowdown in token generation by moving just the experts to the CPU.

I suggest figuring out what your configuration problem is.

Which llama.cpp flags are you using, because I am absolutely not having the same bug you are.

Re: Qwen3-Next

#225

Post by EnPissant »

DiabloD3 wrote: post #224
EnPissant wrote: post #184

Earlier quoted context omitted.

The slowdown is far more than 15% for token generation. Token generation is mostly bottlenecked by memory bandwidth. Dual channel DDR5-6000 has 96GB/s and A rtx 5090 has 1.8TB/s. See my other comment when I show 5x slowdown in token generation by moving just the experts to the CPU.

I suggest figuring out what your configuration problem is. Which llama.cpp flags are you using, because I am absolutely not having the same bug you are.

It's not a bug. It's the reality of token generation. It's bottlenecked by memory bandwidth.

Please publish your own benchmarks proving me wrong.

Re: Qwen3-Next

#226

Post by Zacharias030 »

stingraycharles wrote: post #108
jychang wrote: post #25

Earlier quoted context omitted.

Speculative decoding! It makes inference a LOT faster. Instead of generating tokens one at a time, you generate the second one as well, and then use speculative decoding on that second token (instead of having it be produced by a draft model like Qwen 0.6b). If the token is checked and is correct, then the 2nd token gets generated MUCH faster. If it's wrong, you have to generate it again the normal way (a lot slower…

Because then the second token only needs to be checked, not generated, as it’s already generated? And it’s much faster to generate multiple tokens at the same time than one at a time? Is that the idea? I’m not an expert on LLMs, just a user.

yes, if you know the sequence of tokens ahead of time you can verify them about as quickly as you can generate one more token because of the parallelism benefits.

If you don’t know the future tokens though, then you can’t, and blind guessing of tokens is infeasible because the vocabulary contains circa 100k possible different tokens.

Re: Qwen3-Next

#227

Post by Zacharias030 »

cubefox wrote: post #137
puilp0502 wrote: post #17

Earlier quoted context omitted.

What kind of benefit does Multi-Token Prediction bring to the inference side? Is it only relevant in pretraining efficiency?

> What kind of benefit does Multi-Token Prediction bring to the inference side? Is it only relevant in pretraining efficiency? It is only useful for inference and doesn't help with pretraining. Which actually points to speculative decoding not being sufficiently general, as the same underlying property (some sequences of tokens are easy to predict) could be exploited for training as well. See here: https://goombalab.…

There is no reason that it couldn’t be beneficial for training though.

Re: Qwen3-Next

#228

Post by furyofantares »

EnPissant wrote: post #223
furyofantares wrote: post #221

Earlier quoted context omitted.

I tried a few things and checked CPU usage in Task Manager to see how much work the CPU is doing. KV Cache in GPU and 36/36 layers in GPU: CPU usage under 3%. KV Cache in GPU and 35/36 layers in GPU: CPU usage at 35%. KV Cache moved to CPU and 36/36 layers in GPU: CPU usage at 34%. I believe you that it doesn't make sense to do it this way, it is slower, but it doesn't appear to be doing much of anything on the CPU.…

gpt-oss-120b chooses 4 experts per token and combines them. I don't know how lmstudio works. I only know the fundamentals. There is not way it's sending experts to the GPU per token. Also, the CPU doesn't have much work to do. It's mostly waiting on memory.

> There is not way it's sending experts to the GPU per token.

Right, it seems like either experts are stable across sequential tokens fairly often, or there's more than 4 experts in memory and it's stable within the in-memory experts for sequential tokens fairly often, like the poster said.

Re: Qwen3-Next

#229

Post by jwr »

jabart wrote: post #203
jwr wrote: post #44

Hmm. 80B. These days I am on the lookout for new models in the 32B range, since that is what fits and runs comfortably on my MacBook Pro (M4, 64GB). I use ollama every day for spam filtering: gemma3:27b works great, but I use gpt-oss:20b on a daily basis because it's so much faster and comparable in performance.

Can you talk more about how you are using ollama for spam filtering?

I wrote a little thing that connects to my IMAP server (I run my own E-mail), goes through the unread E-mails in the inbox, processes them (process MIME multipart, extract HTML, describe images and links, etc) and feeds them to an LLM with a prompt. The LLM decides if the message is spam or not.

It's amazingly accurate.

The interesting thing is that after experimentation I found that it's best if the prompt doesn't describe what is spam. The LLMs are somewhat "intelligent", so the prompt now describes me — who I am, what I do, my interests, etc. It's much more effective and generalizes better to fight new kinds of spam.

And a nice side observation is that this kind of system requires no training (so I no longer collect samples of spam) and can't be gamed, because it describes me instead of describing specific kinds of spam.

I have to write it up in a blog post.

Re: Qwen3-Next

#230

Post by jwr »

bigyabai wrote: post #172
jwr wrote: post #44

Hmm. 80B. These days I am on the lookout for new models in the 32B range, since that is what fits and runs comfortably on my MacBook Pro (M4, 64GB). I use ollama every day for spam filtering: gemma3:27b works great, but I use gpt-oss:20b on a daily basis because it's so much faster and comparable in performance.

The model is 80b parameters, but only 3b are activated during inference. I'm running the old 2507 Qwen3 30B model on my 8gb Nvidia card and get very usable performance.

I understand that, but whether it's usable depends on whether ollama can load parts of it into memory on my Mac, and how quickly.
Post reply on HN