Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

magazine.sebastianraschka.com

Post reply on HN
Print view View original

71–80 of 99 posts

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#71

Post by codelion »

starchild3001 wrote: post #65

What stood out to me is how much of gpt-oss’s “newness” isn’t about radical architectural departures, but about a careful layering of well-understood optimizations—RoPE, SwiGLU, GQA, MoE—with some slightly unusual choices (tiny sliding-window sizes, few large experts instead of many small ones, per-head attention sinks). The MXFP4 quantization detail might be the sleeper feature here. Getting 20B running on a 16 GB c…

It is by design. OpenAI is not going to reveal any architectural innovation they have made in their own commercial models.

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#72

Post by selcuka »

Mars008 wrote: post #60
selcuka wrote: post #53

Earlier quoted context omitted.

> GPT OSS 20B is a sparse MoE model. This means it only uses a fraction (3.6B) at a time. They compared it to GPT OSS 120B, which activates 5.1B parameters per token. Given the size of the model it's more than fair to compare it to Qwen3 32B.

You call it fair? 32 / 5.1 > 6, it's takes 6 times more to compute each token. Put it other way, Qwen3 32B is 6 times slower than GPT OSS 120B.

We are talking about accuracy, though. I don't see the point of MoE if a 120B MoE model is not as accurate as even a 32B model.

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#73

Post by littlestymaar »

starchild3001 wrote: post #65

What stood out to me is how much of gpt-oss’s “newness” isn’t about radical architectural departures, but about a careful layering of well-understood optimizations—RoPE, SwiGLU, GQA, MoE—with some slightly unusual choices (tiny sliding-window sizes, few large experts instead of many small ones, per-head attention sinks). The MXFP4 quantization detail might be the sleeper feature here. Getting 20B running on a 16 GB c…

> careful layering of well-understood optimizations—RoPE, SwiGLU, GQA, MoE

They basically cloned Qwen3 on that, before adding the few tweaks you mention afterwards.

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#74

Post by littlestymaar »

Mars008 wrote: post #60
selcuka wrote: post #53

Earlier quoted context omitted.

> GPT OSS 20B is a sparse MoE model. This means it only uses a fraction (3.6B) at a time. They compared it to GPT OSS 120B, which activates 5.1B parameters per token. Given the size of the model it's more than fair to compare it to Qwen3 32B.

You call it fair? 32 / 5.1 > 6, it's takes 6 times more to compute each token. Put it other way, Qwen3 32B is 6 times slower than GPT OSS 120B.

I've read many times that MoE models should be comparable to dense models with a number of parameters equal to the geometric mean of the MoE's total number of parameters and active ones.

In the case of gpt-oss 120B that would means sqrt(5*120)=24B.

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#75

Post by NitpickLawyer »

littlestymaar wrote: post #73
starchild3001 wrote: post #65

What stood out to me is how much of gpt-oss’s “newness” isn’t about radical architectural departures, but about a careful layering of well-understood optimizations—RoPE, SwiGLU, GQA, MoE—with some slightly unusual choices (tiny sliding-window sizes, few large experts instead of many small ones, per-head attention sinks). The MXFP4 quantization detail might be the sleeper feature here. Getting 20B running on a 16 GB c…

> careful layering of well-understood optimizations—RoPE, SwiGLU, GQA, MoE They basically cloned Qwen3 on that, before adding the few tweaks you mention afterwards.

> They basically cloned Qwen3 on that

Oh, come on! GPT4 was rumoured to be an MoE well before Qwen even started releasing models. oAI didn't have to "clone" anything.

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#76

Post by diggan »

codelion wrote: post #71
starchild3001 wrote: post #65

What stood out to me is how much of gpt-oss’s “newness” isn’t about radical architectural departures, but about a careful layering of well-understood optimizations—RoPE, SwiGLU, GQA, MoE—with some slightly unusual choices (tiny sliding-window sizes, few large experts instead of many small ones, per-head attention sinks). The MXFP4 quantization detail might be the sleeper feature here. Getting 20B running on a 16 GB c…

It is by design. OpenAI is not going to reveal any architectural innovation they have made in their own commercial models.

Maybe not a architectural innovation, but both the Harmony format and splitting things into system/developer/user messages instead of just system/user messages, are both novel (in the released weights world) and different enough that I'm still in the process of updating my libraries so I can run fair benchmarks...

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#77

Post by sailingparrot »

littlestymaar wrote: post #73
starchild3001 wrote: post #65

What stood out to me is how much of gpt-oss’s “newness” isn’t about radical architectural departures, but about a careful layering of well-understood optimizations—RoPE, SwiGLU, GQA, MoE—with some slightly unusual choices (tiny sliding-window sizes, few large experts instead of many small ones, per-head attention sinks). The MXFP4 quantization detail might be the sleeper feature here. Getting 20B running on a 16 GB c…

> careful layering of well-understood optimizations—RoPE, SwiGLU, GQA, MoE They basically cloned Qwen3 on that, before adding the few tweaks you mention afterwards.

You seem to be conflating when you first heard about those techniques and when they first appeared. None of those techniques were first seen in Qwen, nor this specific combination of techniques.

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#78

Post by yorwba »

bee_rider wrote: post #68
roscas wrote: post #10

Earlier quoted context omitted.

"The word is SAME." for qwen3-coder and it took like 10 seconds to write all the thinking because it writes it down, but processing was 0 seconds. "I need to find a word that starts with S, ends with E, and contains A. Let me think through this step by step: Starts with S Ends with E Contains A somewhere in the middle One word that fits this pattern is "SAGE" - but that only has 4 letters and would be S-A-G-E, which…

This is tangential because the task was to come up with the riddle, not solve it. But, do reasoning models usually do this poorly? It comes up with a valid solution, SAGE, then disqualifies it for incomprehensible reasons. Then it discovers that SAGE works if it “reads it carefully.” But then seems to disqualify it(?), or at least goes to list other words for some reason. Then it comes up with SAME, a word… with exac…

The model isn't explicitly programmed to constantly second-guess itself, but when you do reinforcement learning with verifiable rewards (RLVR) where only the final answer is verified, even completely nonsensical reasoning can accidentally be rewarded if it gives correct results often enough.

E.g. if the model can generate multiple candidate solutions that are all equally likely (or unlikely) to be correct, it doesn't matter whether you stop at the first one or keep going until a random later one. But if the model can pick the correct solution from multiple candidates better than choosing uniformly at random, generating more candidates becomes an advantage, even if it sometimes results in discarding a correct solution in favor of another one.

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#79

Post by panki27 »

roscas wrote: post #9

From my experience, qwen3-coder is way better. I only have gpt-oss:20b installed to make a few more tests but I give it a program to make a summary of what it does and qwen3 just works in a few seconds, while gpt-oss was cancelled after 5 minuts... doing nothing. So I just use qwen3. Fast and great ouput. If for some reason I don't get what I need, I might use search engines or Perplexity. I have a 10GB 3080 and Ryze…

What Qwen3-Coder model are you using? Quantized or not?

Asking because I'm looking for a good model that fits in 12GB VRAM.

Post reply on HN