Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen3-Next

qwen.ai

Post reply on HN
Print view View original

21–30 of 240 posts

Re: Qwen3-Next

#21

Post by rfoo »

puilp0502 wrote: post #17
jychang wrote: post #5

Coolest part of Qwen3-Next, in my opinion, (after the linear attention parts) is that they do MTP without adding another un-embedding matrix. Deepseek R1 also has a MTP layer (layer 61) https://huggingface.co/deepseek-ai/DeepSeek-R1/blob/main/mod... But Deepseek R1 adds embed_tokens and shared_head.head tensors, which are [129280, 7168] or about 2GB in size at FP8. Qwen3-Next doesn't have that: https://huggingface.co…

What kind of benefit does Multi-Token Prediction bring to the inference side? Is it only relevant in pretraining efficiency?

It could be a better draft model than separately trained EAGLE etc for speculative decoding.

Re: Qwen3-Next

#22

Post by yekanchi »

DiabloD3 wrote: post #20
yekanchi wrote: post #18

how much vram it requires?

Thats not a meaningful question. Models can be quantized to fit into much smaller memory requirements, and not all MoE layers (in MoE models) have to be offloaded to VRAM to maintain performance.

i mean 4bit quantized. i can roughly calculate vram for dense models by model size. but i don't know how to do it for MOE models?

Re: Qwen3-Next

#23

Post by dev_hugepages »

irthomasthomas wrote: post #19

llm -m qwen3-next-80b-a3b-thinking "An ASCII of spongebob" Here's a classic ASCII art representation of SpongeBob SquarePants: .------. / o o \ | | | \___/ | \_______/ llm -m chutes/Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \ "An ASCII of spongebob" Here's an ASCII art of SpongeBob SquarePants: .--..--..--..--..--..--. .' \ (`._ (_) _ \ .' | '._) (_) | \ _.')\ .----..--.' / |(_.' | / .-\-. \---. \ 0| | ( O| O) | | | _…

memorized: https://www.asciiart.eu/cartoons/spongebob-squarepants

Re: Qwen3-Next

#24

Post by NitpickLawyer »

yekanchi wrote: post #18

how much vram it requires?

A good rule of thumb is to think that one param is one unit of storage. The "default" unit of storage these days is bf16 (i.e. 16 bits for 1 weight). So for a 80B model that'll be ~160GB of weights. Then you have quantisation, usually in 8bit and 4bit. That means each weight is "stored" in 8bits or 4bits. So for a 80B model that'll be ~80GB in fp8 and ~40GB in fp4/int4.

But in practice you need a bit more than that. You also need some space for context, and then for kv cache, potentially a model graph, etc.

So you'll see in practice that you need 20-50% more RAM than this rule of thumb.

For this model, you'll need anywhere from 50GB (tight) to 200GB (full) RAM. But it also depends how you run it. With MoE models, you can selectively load some experts (parts of the model) in VRAM, while offloading some in RAM. Or you could run it fully on CPU+RAM, since the active parameters are low - 3B. This should work pretty well even on older systems (DDR4).

Re: Qwen3-Next

#25

Post by jychang »

puilp0502 wrote: post #17
jychang wrote: post #5

Coolest part of Qwen3-Next, in my opinion, (after the linear attention parts) is that they do MTP without adding another un-embedding matrix. Deepseek R1 also has a MTP layer (layer 61) https://huggingface.co/deepseek-ai/DeepSeek-R1/blob/main/mod... But Deepseek R1 adds embed_tokens and shared_head.head tensors, which are [129280, 7168] or about 2GB in size at FP8. Qwen3-Next doesn't have that: https://huggingface.co…

What kind of benefit does Multi-Token Prediction bring to the inference side? Is it only relevant in pretraining efficiency?

Speculative decoding! It makes inference a LOT faster.

Instead of generating tokens one at a time, you generate the second one as well, and then use speculative decoding on that second token (instead of having it be produced by a draft model like Qwen 0.6b). If the token is checked and is correct, then the 2nd token gets generated MUCH faster.

If it's wrong, you have to generate it again the normal way (a lot slower than just checking it). Usually, it's correct, so inference is a lot faster.

Re: Qwen3-Next

#26

Post by keyle »

For a model that can run offline, they've nailed how the website can too.

And it appears like it's thinking about it! /s

Re: Qwen3-Next

#27

Post by EnPissant »

yekanchi wrote: post #22
DiabloD3 wrote: post #20

Earlier quoted context omitted.

Thats not a meaningful question. Models can be quantized to fit into much smaller memory requirements, and not all MoE layers (in MoE models) have to be offloaded to VRAM to maintain performance.

i mean 4bit quantized. i can roughly calculate vram for dense models by model size. but i don't know how to do it for MOE models?

MoE models need just as much VRAM as dense models because every token may use a different set of experts. They just run faster.

Re: Qwen3-Next

#29

Post by ginko »

dev_hugepages wrote: post #23
irthomasthomas wrote: post #19

llm -m qwen3-next-80b-a3b-thinking "An ASCII of spongebob" Here's a classic ASCII art representation of SpongeBob SquarePants: .------. / o o \ | | | \___/ | \_______/ llm -m chutes/Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \ "An ASCII of spongebob" Here's an ASCII art of SpongeBob SquarePants: .--..--..--..--..--..--. .' \ (`._ (_) _ \ .' | '._) (_) | \ _.')\ .----..--.' / |(_.' | / .-\-. \---. \ 0| | ( O| O) | | | _…

memorized: https://www.asciiart.eu/cartoons/spongebob-squarepants

Conveniently removed the artist's signature though.

Re: Qwen3-Next

#30

Post by syntaxing »

The craziest part is how far MoE has come thanks to Qwen. This beats all those 72B dense models we’ve had before and runs faster than 14B model depending on how you off load your VRAM and CPU. That’s insane.
Post reply on HN