Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

magazine.sebastianraschka.com

Post reply on HN
Print view View original

61–70 of 99 posts

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#61

Post by kgeist »

Mars008 wrote: post #60
selcuka wrote: post #53

Earlier quoted context omitted.

> GPT OSS 20B is a sparse MoE model. This means it only uses a fraction (3.6B) at a time. They compared it to GPT OSS 120B, which activates 5.1B parameters per token. Given the size of the model it's more than fair to compare it to Qwen3 32B.

You call it fair? 32 / 5.1 > 6, it's takes 6 times more to compute each token. Put it other way, Qwen3 32B is 6 times slower than GPT OSS 120B.

>Qwen3 32B is 6 times slower than GPT OSS 120B.

Only if 120B fits entirely in the GPU. Otherwise, for me, with a consumer GPU that only has 32 GB VRAM, gpt-oss 120B is actually 2 times slower than Qwen3 32B (37 tok/sec vs. 65 tok/sec)

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#62

Post by Mars008 »

overfeed wrote: post #32
storus wrote: post #14

In my tests, GPT-OSS-120B Q8 was close to DeepSeek R1 671B Q16 in solving graduate-level math but much faster with way fewer thinking tokens.

Supporting TFA'd thesis that it's trained to be good at benchmarks.

Is it bad? It was trained on synthetic data with emphasis on coding and scientific thinking. Good on my opinion, that's what it can be used for. Not as universal do it all model.

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#63

Post by kgeist »

cranberryturkey wrote: post #31
7moritz7 wrote: post #3

Qwen3 is substantially better in my local testing. As in, adheres to the prompt better (pretty much exactly for the 32B parameter variant, very impressive) and is more organic sounding. In simplebench gpt-oss (120 bn) flopped hard so it doesn't appear particularly good at logical puzzles either. So presumably, this comes down to... - training technique or data - dimension - lower number of large experts vs higher num…

qwen3 is slow though. i used it. it worked, but it was slow and lacking features.

On my RTX 5090 with llama.cpp:

gpt-oss 120B - 37 tok/sec (with CPU offloading, doesn't fit in the GPU entirely)

Qwen3 32B - 65 tok/sec

Qwen3 30B-A3B - 150 tok/sec

(all at 4-bit)

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#64

Post by mrheosuper »

eurekin wrote: post #48

I'm still in awe that a local 3090 gpu was able to run the qwen3 coder instruct 30b-a3b exl3 q6 and... Was able to create a sample page, tried starting a server, recognising a leftover server was running, killing it (and forced a prompt for my permission), retrying and finding out it's ip for me to open in the browser. This isn't a demo anymore. That's actually very useful help for interns/juniors already.

How did you do your setup ?, right now the only way i know how to run LLM is through LM studio.

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#65

Post by starchild3001 »

What stood out to me is how much of gpt-oss’s “newness” isn’t about radical architectural departures, but about a careful layering of well-understood optimizations—RoPE, SwiGLU, GQA, MoE—with some slightly unusual choices (tiny sliding-window sizes, few large experts instead of many small ones, per-head attention sinks).

The MXFP4 quantization detail might be the sleeper feature here. Getting 20B running on a 16 GB consumer card, or 120B on a single H100/MI300X without multi-GPU orchestration headaches, could be a bigger enabler for indie devs and researchers than raw benchmark deltas. A lot of experimentation never happens simply because the friction of getting the model loaded is too high.

One open question I’m curious about: given gpt-oss’s design bias toward reasoning (and away from encyclopedic recall), will we start seeing a formal split in open-weight model development—specialized “reasoners” that rely on tool use for facts, and “knowledge bases” tuned for retrieval-heavy work? That separation could change how we architect systems that wrap these models.

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#66

Post by eurekin »

mrheosuper wrote: post #64
eurekin wrote: post #48

I'm still in awe that a local 3090 gpu was able to run the qwen3 coder instruct 30b-a3b exl3 q6 and... Was able to create a sample page, tried starting a server, recognising a leftover server was running, killing it (and forced a prompt for my permission), retrying and finding out it's ip for me to open in the browser. This isn't a demo anymore. That's actually very useful help for interns/juniors already.

How did you do your setup ?, right now the only way i know how to run LLM is through LM studio.

Using tabbyApi and exLlamav2.

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#67

Post by vintermann »

ahmedfromtunis wrote: post #49

When I visit the site I get the error "Your connection is not private". Also: "You cannot visit magazine.sebastianraschka.com right now because the website uses HSTS." Chrome latest on Ubuntu.

First suspicion is that HSTS is doing what it's supposed to, and that you're connecting from somewhere they try to insert themselves in the middle of all https traffic. Https snooping is sadly not uncommon, some businesses think they're entitled to do it for you using their network.

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#68

Post by bee_rider »

roscas wrote: post #10
unstatusthequo wrote: post #7

Earlier quoted context omitted.

Yes. I tried to ask oss-gpt to ask me a riddle. The response was absurd. Came up with a nonsensical question, then told me the answer. The answer was a four letter “word” that wasn’t actually a real word. “What is the word that starts with S, ends with E, and contains A? → SAEA” Then when I said that’s not a word and you gave me the answer already, no fun, it said “I do not have access to confirm that word.”

"The word is SAME." for qwen3-coder and it took like 10 seconds to write all the thinking because it writes it down, but processing was 0 seconds. "I need to find a word that starts with S, ends with E, and contains A. Let me think through this step by step: Starts with S Ends with E Contains A somewhere in the middle One word that fits this pattern is "SAGE" - but that only has 4 letters and would be S-A-G-E, which…

This is tangential because the task was to come up with the riddle, not solve it.

But, do reasoning models usually do this poorly?

It comes up with a valid solution, SAGE, then disqualifies it for incomprehensible reasons.

Then it discovers that SAGE works if it “reads it carefully.” But then seems to disqualify it(?), or at least goes to list other words for some reason.

Then it comes up with SAME, a word… with exactly the same shape as SAGE, just swapped out the irrelevant letter.

What is going on here? Is it programmed to constantly second-guess itself to make it better at finding weaknesses to its answers to harder riddles? But since it doesn’t know how to accept a good answer, it seems like it is just rolling the dice and then stopping at a random point.

I guess it is technically right, but the logic is a total mess.

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#69

Post by fspeech »

faangguyindia wrote: post #57
jszymborski wrote: post #4

Earlier quoted context omitted.

If I had to make a guess, I'd say this has much, much less to do with the architecture and far more to do with the data and training pipeline. Many have speculated that gpt-oss has adopted a Phi-like synthetic-only dataset and focused mostly on gaming metrics, and I've found the evidence so far to be sufficiently compelling.

this is exactly why strongest model gonna lose out to weaker models if the later ones have more data for example, i was using deep seek webui and getting decent on point answers but it simply does not have latest data. So, while Deep Seek R1 might be better model than Grok3 or even Grok4, it not having access to "twitter data" basically puts it behind. Same is case with OpenAI, if OpenAI has access to fast data from…

On the open source library part, you can ask DeepWiki the questions yourself and feed the answers to the LLMs by hand. DeepWiki gives you high quality answers because they are grounded in code and you can check the veracity yourself.

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#70

Post by asabla »

starchild3001 wrote: post #65

What stood out to me is how much of gpt-oss’s “newness” isn’t about radical architectural departures, but about a careful layering of well-understood optimizations—RoPE, SwiGLU, GQA, MoE—with some slightly unusual choices (tiny sliding-window sizes, few large experts instead of many small ones, per-head attention sinks). The MXFP4 quantization detail might be the sleeper feature here. Getting 20B running on a 16 GB c…

> that rely on tool use for facts, and “knowledge bases” tuned for retrieval-heavy work

I would say this isn't exclusive to the smaller OSS models. But rather a trait of Openai's models all together now.

This becomes especially apparent with the introduction of GPT-5 in ChatGPT. Their focus on routing your request to different modes and searching the web automatically (relying on an Agentic workflows in the background) is probably key to the overall quality of the output.

So far, it's quite easy to get their OSS models to follow instructions reliably. Qwen models has been pretty decent at this too for some time now.

I think if we give it another generation or two, we're at the point of having compotent enough models to start running more advanced agentic workflows. On modest hardware. We're almost there now, but not quite yet

Post reply on HN