Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen3-Next

qwen.ai

Post reply on HN
Print view View original

41–50 of 240 posts

Re: Qwen3-Next

#41

Post by EnPissant »

regularfry wrote: post #39
EnPissant wrote: post #27

Earlier quoted context omitted.

MoE models need just as much VRAM as dense models because every token may use a different set of experts. They just run faster.

This isn't quite right: it'll run with the full model loaded to RAM, swapping in the experts as it needs. It has turned out in the past that experts can be stable across more than one token so you're not swapping as much as you'd think. I don't know if that's been confirmed to still be true on recent MoEs, but I wouldn't be surprised.

What you are describing would be uselessly slow and nobody does that.

Re: Qwen3-Next

#42

Post by moffkalast »

jychang wrote: post #25
puilp0502 wrote: post #17

Earlier quoted context omitted.

What kind of benefit does Multi-Token Prediction bring to the inference side? Is it only relevant in pretraining efficiency?

Speculative decoding! It makes inference a LOT faster. Instead of generating tokens one at a time, you generate the second one as well, and then use speculative decoding on that second token (instead of having it be produced by a draft model like Qwen 0.6b). If the token is checked and is correct, then the 2nd token gets generated MUCH faster. If it's wrong, you have to generate it again the normal way (a lot slower…

Hmm but isn't the checking only required because the draft model is not the same model and can only speculate what the main one is thinking, hence the name? If the main model generates two tokens itself, then how can it be wrong about its own predictions?

Re: Qwen3-Next

#43

Post by moffkalast »

syntaxing wrote: post #30

The craziest part is how far MoE has come thanks to Qwen. This beats all those 72B dense models we’ve had before and runs faster than 14B model depending on how you off load your VRAM and CPU. That’s insane.

In retrospect it's actually funny that last year Meta spent so many resources training a dense 405B model that both underperforms compared to models a tenth its size and is impossible to run at a reasonable speed on any hardware in existence.

Re: Qwen3-Next

#44

Post by jwr »

Hmm. 80B. These days I am on the lookout for new models in the 32B range, since that is what fits and runs comfortably on my MacBook Pro (M4, 64GB).

I use ollama every day for spam filtering: gemma3:27b works great, but I use gpt-oss:20b on a daily basis because it's so much faster and comparable in performance.

Re: Qwen3-Next

#45

Post by SonOfLilit »

moffkalast wrote: post #42
jychang wrote: post #25

Earlier quoted context omitted.

Speculative decoding! It makes inference a LOT faster. Instead of generating tokens one at a time, you generate the second one as well, and then use speculative decoding on that second token (instead of having it be produced by a draft model like Qwen 0.6b). If the token is checked and is correct, then the 2nd token gets generated MUCH faster. If it's wrong, you have to generate it again the normal way (a lot slower…

Hmm but isn't the checking only required because the draft model is not the same model and can only speculate what the main one is thinking, hence the name? If the main model generates two tokens itself, then how can it be wrong about its own predictions?

If you ask me to guess an answer, I'll _usually_ produce the same answer as if I had time to think about it deeply, but not always...

Re: Qwen3-Next

#46

Post by theanonymousone »

NitpickLawyer wrote: post #24
yekanchi wrote: post #18

how much vram it requires?

A good rule of thumb is to think that one param is one unit of storage. The "default" unit of storage these days is bf16 (i.e. 16 bits for 1 weight). So for a 80B model that'll be ~160GB of weights. Then you have quantisation, usually in 8bit and 4bit. That means each weight is "stored" in 8bits or 4bits. So for a 80B model that'll be ~80GB in fp8 and ~40GB in fp4/int4. But in practice you need a bit more than that.…

But the RAM+VRAM can never be less than the size of the total (not active) model, right?

Re: Qwen3-Next

#47

Post by NitpickLawyer »

theanonymousone wrote: post #46
NitpickLawyer wrote: post #24

Earlier quoted context omitted.

A good rule of thumb is to think that one param is one unit of storage. The "default" unit of storage these days is bf16 (i.e. 16 bits for 1 weight). So for a 80B model that'll be ~160GB of weights. Then you have quantisation, usually in 8bit and 4bit. That means each weight is "stored" in 8bits or 4bits. So for a 80B model that'll be ~80GB in fp8 and ~40GB in fp4/int4. But in practice you need a bit more than that.…

But the RAM+VRAM can never be less than the size of the total (not active) model, right?

Correct. You want everything loaded, but for each forward pass just some experts get activated so the computation is less than in a dense model.

That being said, there are libraries that can load a model layer by layer (say from an ssd) and technically perform inference with ~8gb of RAM, but it'd be really really slow.

Re: Qwen3-Next

#48

Post by mlvljr »

irthomasthomas wrote: post #35
ginko wrote: post #29

Earlier quoted context omitted.

Conveniently removed the artist's signature though.

Yes - they all do that. Actually, most attempts start well but unravel toward the end. llm -m chutes/Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \ "An ASCII of spongebob" Here's an ASCII art of SpongeBob SquarePants: ``` .--..--..--..--..--..--. .' \ (`._ (_) _ \ .' | '._) (_) | \ _.')\ .----..--. / |(_.' | / .-\-. \ \ 0| | ( O| O) | | _ | .--.____.'._.-. /.' ) | (_.' .-'"`-. _.-._.-.--.-. / .''. | .' `-. .-'-. .-'"`-.`-…

Going through shredder

Re: Qwen3-Next

#49

Post by cbm-vic-20 »

irthomasthomas wrote: post #35
ginko wrote: post #29

Earlier quoted context omitted.

Conveniently removed the artist's signature though.

Yes - they all do that. Actually, most attempts start well but unravel toward the end. llm -m chutes/Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \ "An ASCII of spongebob" Here's an ASCII art of SpongeBob SquarePants: ``` .--..--..--..--..--..--. .' \ (`._ (_) _ \ .' | '._) (_) | \ _.')\ .----..--. / |(_.' | / .-\-. \ \ 0| | ( O| O) | | _ | .--.____.'._.-. /.' ) | (_.' .-'"`-. _.-._.-.--.-. / .''. | .' `-. .-'-. .-'"`-.`-…

Ph'nglui mglw'nafh Cthulhu R'lyeh wgah'nagl fhtagn.

Re: Qwen3-Next

#50

Post by [deleted] »

irthomasthomas wrote: post #19

llm -m qwen3-next-80b-a3b-thinking "An ASCII of spongebob" Here's a classic ASCII art representation of SpongeBob SquarePants: .------. / o o \ | | | \___/ | \_______/ llm -m chutes/Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \ "An ASCII of spongebob" Here's an ASCII art of SpongeBob SquarePants: .--..--..--..--..--..--. .' \ (`._ (_) _ \ .' | '._) (_) | \ _.')\ .----..--.' / |(_.' | / .-\-. \---. \ 0| | ( O| O) | | | _…

[deleted]
Post reply on HN