Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

magazine.sebastianraschka.com

Post reply on HN
Print view View original

91–99 of 99 posts

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#91

Post by energy123 »

poorman wrote: post #37
oezi wrote: post #36

One question I was wondering about regarding the open models released by big labs is how much more the could improve with additional training. GPT-OSS has 2.1m hours of training, how much score improvements could we see at double that?

As we saw with GPT-5 the RL technique of training doesn't scale forever

Unless GPT-5 is 30% cheaper to run than o3. Then it's scaling brilliantly given the small gap between release dates. People are really drawing too many conclusions from too little information.

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#92

Post by selcuka »

littlestymaar wrote: post #74
Mars008 wrote: post #60

Earlier quoted context omitted.

You call it fair? 32 / 5.1 > 6, it's takes 6 times more to compute each token. Put it other way, Qwen3 32B is 6 times slower than GPT OSS 120B.

I've read many times that MoE models should be comparable to dense models with a number of parameters equal to the geometric mean of the MoE's total number of parameters and active ones. In the case of gpt-oss 120B that would means sqrt(5*120)=24B.

> In the case of gpt-oss 120B that would means sqrt(5*120)=24B.

That's actually in line with what I had (unscientifically) expected. Claude Sonnet 4 seems to agree:

> The most accurate approach for your specific 120B MoE (5.1B active) would be to test it empirically against dense models in the 10-30B range.

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#93

Post by wickedsight »

eurekin wrote: post #84
faangguyindia wrote: post #56

Earlier quoted context omitted.

yesterday, i signed up for qwen3-coder-plus. It fails 4/10 "diff" edit format in various code editing tools i use. Gemini Pro 2.5 with diff fenced edit format, rarely fails. So i don't see this Qwen3 hype unless i am using wrong edit format, can anyone tell me which edit format will work better with Qwen3? https://aider.chat/docs/more/edit-formats.html

I'm running a30-a3b-instruct q6 quant on exllamav2 and checked few simple tasks in roo and cline. Prompt adherence, tool calling and file changing worked flawlessly

Maybe I'm doing something wrong, but in my testing with Roo and Qwen3-Coder-30B via MLX, it constantly ends up in loops and often doesn't manage to finish editing a file, leaving it half finished.

If I give it really simple, straight forward tasks it works quite nice though.

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#94

Post by mike_hearn »

I'm really not a PyTorch expert so this is most likely a newbie error, but could someone explain to me the code in Figure 7?

The code circled as "4 x emb_dim" doesn't seem to apply a 4x multiplier anywhere. Actually, the layer definitions of fc1 and fc2 in the SwiGLU variant appear to be identical to the code in the regular feed forward block. What is making the two layers in the second code snippet different sizes to fc1 in the first?

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#95

Post by bee_rider »

omneity wrote: post #52
7moritz7 wrote: post #3

Qwen3 is substantially better in my local testing. As in, adheres to the prompt better (pretty much exactly for the 32B parameter variant, very impressive) and is more organic sounding. In simplebench gpt-oss (120 bn) flopped hard so it doesn't appear particularly good at logical puzzles either. So presumably, this comes down to... - training technique or data - dimension - lower number of large experts vs higher num…

Qwen3 32B is a dense model, it uses all its parameters all the time. GPT OSS 20B is a sparse MoE model. This means it only uses a fraction (3.6B) at a time. It’s a tradeoff that makes it faster to run than a dense 20B model and much smarter than a 3.6B one. In practice the fairest comparison would be to a dense ~8B model. Qwen Coder 30B A3B is a good sparse comparison point as well.

Tangential question from an outsider:

When people talk about sparse or dense models, are they spare or dense matrices in the conventional numerical linear algebra sense? (Something like a csr matrix?)

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#96

Post by fdalvi »

mike_hearn wrote: post #94

I'm really not a PyTorch expert so this is most likely a newbie error, but could someone explain to me the code in Figure 7? The code circled as "4 x emb_dim" doesn't seem to apply a 4x multiplier anywhere. Actually, the layer definitions of fc1 and fc2 in the SwiGLU variant appear to be identical to the code in the regular feed forward block. What is making the two layers in the second code snippet different sizes t…

It is indeed not something clarified by the code snippets; In normal feedforward layers, it is common to choose the "hidden_dim = 4 x emb_dim", while in GLU feedforward layer, the convention is to use "hidden_dim = 2/3 * regular_ffn_hidden_dim" (to keep the overall number of parameters roughly the same). In the case of gpt-oss, they chose to go a bit more extreme and set "hidden_dim = emb_dim", thus reducing the overall number of parameters!

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#97

Post by mike_hearn »

fdalvi wrote: post #96
mike_hearn wrote: post #94

I'm really not a PyTorch expert so this is most likely a newbie error, but could someone explain to me the code in Figure 7? The code circled as "4 x emb_dim" doesn't seem to apply a 4x multiplier anywhere. Actually, the layer definitions of fc1 and fc2 in the SwiGLU variant appear to be identical to the code in the regular feed forward block. What is making the two layers in the second code snippet different sizes t…

It is indeed not something clarified by the code snippets; In normal feedforward layers, it is common to choose the "hidden_dim = 4 x emb_dim", while in GLU feedforward layer, the convention is to use "hidden_dim = 2/3 * regular_ffn_hidden_dim" (to keep the overall number of parameters roughly the same). In the case of gpt-oss, they chose to go a bit more extreme and set "hidden_dim = emb_dim", thus reducing the over…

Ah, thank you!

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#98

Post by Mars008 »

littlestymaar wrote: post #74
Mars008 wrote: post #60

Earlier quoted context omitted.

You call it fair? 32 / 5.1 > 6, it's takes 6 times more to compute each token. Put it other way, Qwen3 32B is 6 times slower than GPT OSS 120B.

I've read many times that MoE models should be comparable to dense models with a number of parameters equal to the geometric mean of the MoE's total number of parameters and active ones. In the case of gpt-oss 120B that would means sqrt(5*120)=24B.

Not sure there is on formula. Because there are two different cases:

1) performance constrained. like NVidia Spark with 128GB or AGX with 64GB.

2) memory constrained. like consumers' GPUs.

In first case MoE is clear win. They fit and run faster. In second case dense models will produce better results. And if performance in token/sec is acceptable then they are better choice.

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#99

Post by chaos_emergent »

typon wrote: post #39
chaos_emergent wrote: post #38

> This is likely because LLMs are typically trained for only a single epoch over massive datasets, which is in contrast to the multi-hundred-epoch training regimes for which dropout was first introduced. Wait, is this true? That seems like a wild statement to make, relatively unsubstantiated?

No this is well known. Look for Table 2.2 in GPT3 paper.

Thank you, that was a wild thing to learn!
Post reply on HN