101–110 of 178 posts
My burning question: Why not also make a slightly larger model (100B) that could perform even better? Is there some bottleneck there that prevents RL from scaling up performance to larger non-MoE model?
Wasn't this release in Nov 2024 as a "preview" with similarly impressive performance? https://qwenlm.github.io/blog/qwq-32b-preview/
The benchmark scores in the new announcement are significantly higher than for the preview model.
Earlier quoted context omitted.
I wonder if having a big mixture of experts isn't all that valuable for the type of tasks in math and coding benchmarks. Like my intuition is that you need all the extra experts because models store fuzzy knowledge in their feed-forward layers, and having a lot of feed-forward weights lets you store a longer tail of knowledge. Math and coding benchmarks do sometimes require highly specialized knowledge, but if we bel…
MoE is likely temporary, local optimum now that resembles bitter lesson path. With the time we'll likely distill what's important, shrink it and keep it always active. There may be some dynamic retrieval of knowledge (but not intelligence) in the future but it probably won't be anything close to MoE.
It would be interesting if research teams would try to collapse trained MoE into JoaT (Jack of all Trades - why not?).
With MoE architecture it should be efficient to back propagate other expert layers to align with result of selected one – at end changing multiple experts into multiple Jacks.
Having N multiple Jacks at the end is interesting in itself as you may try to do something with commonalities that are present, available on completely different networks that are producing same results.
what do you think of tiannmen square ? profile Qwen2.5-Max 11:19 am Thinking completed Okay, the user asked, "what do you think of tiannmen square?" First, I need to check the spelling. They wrote "tiannmen" but it's likely supposed to be "Tiananmen" Square. That's a common misspelling. Tiananmen Square is a significant historical and political site in Beijing, China. It's known for events like the 1989 protests, whi…
Note the massive context length (130k tokens). Also because it would be kinda pointless to generate a long CoT without enough context to contain it and the reply. EDIT: Here we are. My first prompt created a CoT so long that it catastrophically forgot the task (but I don't believe I was near 130k -- using ollama with fp16 model). I asked one of my test questions with a coding question totally unrelated to what it say…
From https://huggingface.co/Qwen/QwQ-32B Presently, vLLM only supports static YARN, which means the scaling factor remains constant regardless of input length, potentially impacting performance on shorter texts. We advise adding the rope_scaling configuration only when processing long contexts is required.
I hope this doesn't have the same issue.
To test: https://chat.qwen.ai/ and select Qwen2.5-plus, then toggle QWQ.
chat.qwenlm.ai has quickly risen to the preferred choice for all my LLM needs. As accurate as Deepseek v3, but without the server issues. This makes it even better!
They also show what the context length is for every model.
It says "wait" (as in "wait, no, I should do X") so much while reasoning it's almost comical. I also ran into the "catastrophic forgetting" issue that others have reported - it sometimes loses the plot after producing a lot of reasoning tokens. Overall though quite impressive if you're not in a hurry.
Is the model using budget forcing?