Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen3-Next

qwen.ai

Post reply on HN
Print view View original

91–100 of 240 posts

Re: Qwen3-Next

#91

Post by EMM_386 »

moffkalast wrote: post #42
jychang wrote: post #25

Earlier quoted context omitted.

Speculative decoding! It makes inference a LOT faster. Instead of generating tokens one at a time, you generate the second one as well, and then use speculative decoding on that second token (instead of having it be produced by a draft model like Qwen 0.6b). If the token is checked and is correct, then the 2nd token gets generated MUCH faster. If it's wrong, you have to generate it again the normal way (a lot slower…

Hmm but isn't the checking only required because the draft model is not the same model and can only speculate what the main one is thinking, hence the name? If the main model generates two tokens itself, then how can it be wrong about its own predictions?

I believe it's something along these lines. The MTP head runs simultaneously and generates a probability list based on what it thinks the results will be, learned during training.

If n+1 = "Barack" then n+2 = "Obama" (confidence: 0.90) If n+1 = "The" then n+2 = "quick" (confidence: 0.45) If n+1 = "President" then n+2 = "Biden" (confidence: 0.75)

A threshold is set (say, as 90%) so that if the n+2 prediction is above that (as in the first example) it uses it without having to determine it with the main model. It's confident "enough".

Re: Qwen3-Next

#92

Post by pilotneko »

Alifatisk wrote: post #65

Alibaba keeps releasing gold content I just tried Qwen3-Next-80B-A3B on Qwen chat, and it's fast! The quality seem to match Qwen3-235B-A22B. Quite impressive how they achieved this. Can't wait for the benchmarks at Artificial analysis According to Qwen Chat, Qwen3-Next has the following limits: Maximum context length: 262,144 tokens Max summary generation length: 32,768 tokens This is 2x higher on context length and…

If you read the model card, Qwen3-Next can be extended to 1M context length with YaRN.

> Qwen3-Next natively supports context lengths of up to 262,144 tokens. For conversations where the total length (including both input and output) significantly exceeds this limit, we recommend using RoPE scaling techniques to handle long texts effectively. We have validated the model's performance on context lengths of up to 1 million tokens using the YaRN method.

Source: https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct#proc...

Re: Qwen3-Next

#93

Post by skirmish »

theanonymousone wrote: post #52
NitpickLawyer wrote: post #47

Earlier quoted context omitted.

Correct. You want everything loaded, but for each forward pass just some experts get activated so the computation is less than in a dense model. That being said, there are libraries that can load a model layer by layer (say from an ssd) and technically perform inference with ~8gb of RAM, but it'd be really really slow.

Can you give me a name please? Is that distributed llama or something else?

I have not used it but this is probably it: https://github.com/lyogavin/airllm

Re: Qwen3-Next

#94

Post by wickedsight »

Razengan wrote: post #80
jychang wrote: post #5

Coolest part of Qwen3-Next, in my opinion, (after the linear attention parts) is that they do MTP without adding another un-embedding matrix. Deepseek R1 also has a MTP layer (layer 61) https://huggingface.co/deepseek-ai/DeepSeek-R1/blob/main/mod... But Deepseek R1 adds embed_tokens and shared_head.head tensors, which are [129280, 7168] or about 2GB in size at FP8. Qwen3-Next doesn't have that: https://huggingface.co…

Could someone kindly point to a convenient all-on-one ELI5 of all these words? :')

For me, ChatGPT or any of the other current thinking models are very useful for this type of stuff. I just ask to explain it on my level and then I can ask questions for clarification.

Re: Qwen3-Next

#95

Post by Leynos »

nakamoto_damacy wrote: post #64
matchcc wrote: post #61

Earlier quoted context omitted.

I think there is some distillation relationship between Kimi K2 and Qwen Coder or other related other models, or same training data. I tried most of LLMs, only kimi K2 gave the exact same ASCII. kimi K2: Here’s a classic ASCII art of SpongeBob SquarePants for you: .--..--..--..--..--..--. .' \ (`._ (_) _ \ .' | '._) (_) | \ _.')\ .----..---. / |(_.' | / .-\-. \ | \ 0| | ( O| O) | o| | _ | .--.____.'._.-. | \ (_) | o…

For ascii to look right, not messed up, the generator has to know the width of the div in ascii characters, e.g. 80, 240, etc, so it can make sure the lines don't wrap. So how does an LLM know anything about the UI it's serving? Is it just luck? what if you ask it to draw something that like 16:9 in aspect ratio... would it know to scale it dowm so lines won't wrap? how about loss of details if it does? Also, is it a…

They don't see runs of spaces very well, so most of them are terrible at ASCII art. (They'll often regurgitate something from their training data rather than try themselves.)

And unless their terminal details are included in the context, they'll just have to guess.

Re: Qwen3-Next

#96

Post by mmmllm »

amelius wrote: post #87
mmmllm wrote: post #82

Earlier quoted context omitted.

Sure but where is the demand going to come from? LLMs are already in every google search, in Whatsapp/Messenger, throughout Google workspace, Notion, Slack, etc. ChatGPT already has a billion users. Plus penetration is already very high in the areas where they are objectively useful: programming, customer care etc. I just don't see where the 100-1000x demand comes from to offset this. Would be happy to hear other vie…

If you can make an LLM solve a problem but from 100 different angles at the same time, that's worth something.

Isn't that essentially how the MoE models already work? Besides, if that were infinitely scalable, wouldn't we have a subset of super-smart models already at very high cost?

Besides, this would only apply for very few use cases. For a lot of basic customer care work, programming, quick research, I would say LLMs are already quite good without running it 100X.

Re: Qwen3-Next

#97

Post by philipp-gayret »

mmmllm wrote: post #82
amelius wrote: post #70

Earlier quoted context omitted.

https://en.wikipedia.org/wiki/Jevons_paradox

Sure but where is the demand going to come from? LLMs are already in every google search, in Whatsapp/Messenger, throughout Google workspace, Notion, Slack, etc. ChatGPT already has a billion users. Plus penetration is already very high in the areas where they are objectively useful: programming, customer care etc. I just don't see where the 100-1000x demand comes from to offset this. Would be happy to hear other vie…

If LLMs were next to free and faster I would personally increase my consumption 100x or more and Im only the "programming" category.

Re: Qwen3-Next

#98

Post by CuriouslyC »

Jgoauh wrote: post #2

Seems impressive, i believe better architectures are really the path forward, i don't think you need more than 100B params taking this model and what GPT OSS 120B can acchieve

We definitely need more parameters, low param models are hallucination machines, though low actives is probably fine assuming the routing is good.

Re: Qwen3-Next

#99

Post by CuriouslyC »

NitpickLawyer wrote: post #11
vintermann wrote: post #8

Earlier quoted context omitted.

Could you give some practical examples? I don't know what Qwen's 36T-token training set is like, so I don't know what it's overfitting to...

Take math and coding for example: - in math, if they can solve a problem, or a class of problems, they'll solve it. If you use a "thinking" model + maj@x, you'll get strong results. But if you try for example to have the model consider a particular way or method of exploring a problem, it'll default to "solving" mode. It's near impossible to have it do something else with a math problem, other than solving it. Say "e…

That's the thing people miss that's so good about GPT5. It's incredibly steerable in a way a lot of models aren't.

Re: Qwen3-Next

#100

Post by mirekrusin »

mmmllm wrote: post #96
amelius wrote: post #87

Earlier quoted context omitted.

If you can make an LLM solve a problem but from 100 different angles at the same time, that's worth something.

Isn't that essentially how the MoE models already work? Besides, if that were infinitely scalable, wouldn't we have a subset of super-smart models already at very high cost? Besides, this would only apply for very few use cases. For a lot of basic customer care work, programming, quick research, I would say LLMs are already quite good without running it 100X.

MoE is something different - it's a technique to activate just a small subset of parameters during inference.

Whatever is good enough now, can be much better for the same cost (time, computation, actual cost). People will always choose better over worse.

Post reply on HN