GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2 • Hacker News Board

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#31

Post by cranberryturkey » Sun, Aug 10, 2025, 8:09 PM UTC

7moritz7 wrote: post #3

Qwen3 is substantially better in my local testing. As in, adheres to the prompt better (pretty much exactly for the 32B parameter variant, very impressive) and is more organic sounding. In simplebench gpt-oss (120 bn) flopped hard so it doesn't appear particularly good at logical puzzles either. So presumably, this comes down to... - training technique or data - dimension - lower number of large experts vs higher num…

qwen3 is slow though. i used it. it worked, but it was slow and lacking features.

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#32

Post by overfeed » Sun, Aug 10, 2025, 8:09 PM UTC

storus wrote: post #14

In my tests, GPT-OSS-120B Q8 was close to DeepSeek R1 671B Q16 in solving graduate-level math but much faster with way fewer thinking tokens.

Supporting TFA'd thesis that it's trained to be good at benchmarks.

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#33

Post by indigodaddy » Sun, Aug 10, 2025, 8:18 PM UTC

lvl155 wrote: post #26

roscas wrote: post #9

From my experience, qwen3-coder is way better. I only have gpt-oss:20b installed to make a few more tests but I give it a program to make a summary of what it does and qwen3 just works in a few seconds, while gpt-oss was cancelled after 5 minuts... doing nothing. So I just use qwen3. Fast and great ouput. If for some reason I don't get what I need, I might use search engines or Perplexity. I have a 10GB 3080 and Ryze…

Qwen3 coder 480B is quite good and on par with Sonnet 4. It’s the first time I realized the Chinese models are probably going to eclipse US-based models pretty soon, at least for coding.

Where do you use qwen3 480b from, I'm not even seeing it on Openrouter. EDIT nm, openrouter is just calling it qwen3-coder-- when I click for more info it shows it's Qwen3-Coder-480B-A35B-Instruct. And it's one of their free models. Nice

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#34

Post by gglon » Sun, Aug 10, 2025, 8:34 PM UTC

> At the time of writing, the highest-ranking non-purely-transformer-based model on the LM Arena is Jamba, which is a transformer–state space model hybrid, at rank 96.)

Tencent's hunyuan-turbos, another hybrid, is currently ranked at 22. https://arxiv.org/abs/2505.15431

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#35

Post by threeducks » Sun, Aug 10, 2025, 8:38 PM UTC

yunusabd wrote: post #24

threeducks wrote: post #18

Earlier quoted context omitted.

FWIW, I asked gpt-oss-120b this question 10 times and the answer was always "sauce", "sane" or "sale". I also tried different temperatures (from 0 to 1), which did not seem to have an effect on the correctness of the answer. EDIT: I now have also questioned the smaller gpt-oss-20b (free) 10 times via OpenRouter (default settings, provider was AtlasCloud) and the answers were: sage, sane, sane, space, sane, sane, sane…

GP asked the model to _create_ a riddle, not solve a given one.

Yes, but the odds of getting GPT-OSS to respond with that riddle are pretty low and it is not necessary to demonstrate whether the LLM can answer the riddle correctly.

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#36

Post by oezi » Sun, Aug 10, 2025, 8:45 PM UTC

One question I was wondering about regarding the open models released by big labs is how much more the could improve with additional training. GPT-OSS has 2.1m hours of training, how much score improvements could we see at double that?

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#37

Post by poorman » Sun, Aug 10, 2025, 8:52 PM UTC

oezi wrote: post #36

One question I was wondering about regarding the open models released by big labs is how much more the could improve with additional training. GPT-OSS has 2.1m hours of training, how much score improvements could we see at double that?

As we saw with GPT-5 the RL technique of training doesn't scale forever

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#38

Post by chaos_emergent » Sun, Aug 10, 2025, 8:56 PM UTC

> This is likely because LLMs are typically trained for only a single epoch over massive datasets, which is in contrast to the multi-hundred-epoch training regimes for which dropout was first introduced.

Wait, is this true? That seems like a wild statement to make, relatively unsubstantiated?

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#39

Post by typon » Sun, Aug 10, 2025, 8:57 PM UTC

chaos_emergent wrote: post #38

> This is likely because LLMs are typically trained for only a single epoch over massive datasets, which is in contrast to the multi-hundred-epoch training regimes for which dropout was first introduced. Wait, is this true? That seems like a wild statement to make, relatively unsubstantiated?

No this is well known. Look for Table 2.2 in GPT3 paper.

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#40

Post by poorman » Sun, Aug 10, 2025, 8:59 PM UTC

This article really goes into a lot of detail which is nice. gpt-oss is just not good for agentic use in my observation.

tldr; I'll save you a lot of time trying things out for yourself. If you are on a >=32 GB Mac download LMStudio and then the `qwen3-coder-30b-a3b-instruct-mlx@5bit` model. It uses ~20 GB of RAM so a 32GB machine is plenty. Set it up with opencode [1] and you're off to the races! It has great tool calling ability. The tool calling ability of gpt-oss doesn't even come close in my observations.

[1] https://opencode.ai/