Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

magazine.sebastianraschka.com

Post reply on HN
Print view View original

21–30 of 99 posts

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#21

Post by mhitza »

ModelForge wrote: post #17
mhitza wrote: post #12

Earlier quoted context omitted.

I've been using lightly gpt-oss-20b but what I've found is that for smaller (single sentence) prompts it was easy enough to have it loop infinitely. Since I'm running it with llama.cpp I've set a small repetition penalty and haven't encountered those issues since (I'm using it a couple of times a day to analyze diffs, so I might have just gotten lucky since)

I’ve been using the ollama version (uses about 13 Gb RAM on macOS) and haven’t had that issue yet. I wonder if that’s maybe an issue of the llama.cpp port?

Never used ollama, only ready to go models via llamafile and llama.cpp.

Maybe ollama has some defaults it applies to models? I start testing models at 0 temp and tweak from there depending how they behave.

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#22

Post by indigodaddy »

pryelluw wrote: post #16

The Qwen3 4B has been very good to use local. I barely use the online models. Web searches are now more targeted thanks to it. Don’t quite fully trust the output but it’s generally good. Mods like these will revolutionize local knowledge and automation

Qwen is telling you better search parameters to then search the web with, or qwen is actually doing web searches for you?

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#23

Post by nicolaslem »

mhitza wrote: post #12
roscas wrote: post #9

From my experience, qwen3-coder is way better. I only have gpt-oss:20b installed to make a few more tests but I give it a program to make a summary of what it does and qwen3 just works in a few seconds, while gpt-oss was cancelled after 5 minuts... doing nothing. So I just use qwen3. Fast and great ouput. If for some reason I don't get what I need, I might use search engines or Perplexity. I have a 10GB 3080 and Ryze…

I've been using lightly gpt-oss-20b but what I've found is that for smaller (single sentence) prompts it was easy enough to have it loop infinitely. Since I'm running it with llama.cpp I've set a small repetition penalty and haven't encountered those issues since (I'm using it a couple of times a day to analyze diffs, so I might have just gotten lucky since)

I had the same issue with other models where they would loop repeating the same character, sentence or paragraph indefinitely. Turns out the context size some tools set by default is 2k and this is way too small.

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#24

Post by yunusabd »

threeducks wrote: post #18
unstatusthequo wrote: post #7

Earlier quoted context omitted.

Yes. I tried to ask oss-gpt to ask me a riddle. The response was absurd. Came up with a nonsensical question, then told me the answer. The answer was a four letter “word” that wasn’t actually a real word. “What is the word that starts with S, ends with E, and contains A? → SAEA” Then when I said that’s not a word and you gave me the answer already, no fun, it said “I do not have access to confirm that word.”

FWIW, I asked gpt-oss-120b this question 10 times and the answer was always "sauce", "sane" or "sale". I also tried different temperatures (from 0 to 1), which did not seem to have an effect on the correctness of the answer. EDIT: I now have also questioned the smaller gpt-oss-20b (free) 10 times via OpenRouter (default settings, provider was AtlasCloud) and the answers were: sage, sane, sane, space, sane, sane, sane…

GP asked the model to _create_ a riddle, not solve a given one.

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#25

Post by lvl155 »

mark_l_watson wrote: post #15

Wow, Sebastian Raschk's blog articles are jewels - much appreciated. I use the get-oss and qwen3 models a lot (smaller models locally using Ollama and LM Studio) and commercial APIs for the full size models. For local model use, I get very good results with get-oss when I "over prompt," that is, I specify a larger amount of context information than I usually do. Qwen3 is simply awesome. Until about three years ago, I…

He does an amazing job of keeping me up to date on this insanely fast-paced space.

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#26

Post by lvl155 »

roscas wrote: post #9

From my experience, qwen3-coder is way better. I only have gpt-oss:20b installed to make a few more tests but I give it a program to make a summary of what it does and qwen3 just works in a few seconds, while gpt-oss was cancelled after 5 minuts... doing nothing. So I just use qwen3. Fast and great ouput. If for some reason I don't get what I need, I might use search engines or Perplexity. I have a 10GB 3080 and Ryze…

Qwen3 coder 480B is quite good and on par with Sonnet 4. It’s the first time I realized the Chinese models are probably going to eclipse US-based models pretty soon, at least for coding.

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#27

Post by BoorishBears »

7moritz7 wrote: post #3

Qwen3 is substantially better in my local testing. As in, adheres to the prompt better (pretty much exactly for the 32B parameter variant, very impressive) and is more organic sounding. In simplebench gpt-oss (120 bn) flopped hard so it doesn't appear particularly good at logical puzzles either. So presumably, this comes down to... - training technique or data - dimension - lower number of large experts vs higher num…

MoE expected performance = sqrt(active heads * total parameter count)

sqrt(120*5) ~= 24

GPT-OSS 120B is effectively a 24B parameter model with the speed of a much smaller model

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#28

Post by adastra22 »

CuriouslyC wrote: post #6
7moritz7 wrote: post #5

Earlier quoted context omitted.

That would be interesting. I've been a bit sceptical of the entire strategy from the beginning. If oss was actually as good as o3 mini and in some cases o4 mini outside benchmarks, that would undermine openai's api offer for gpt 5 nano and maybe mini too. Edit: found this analysis, it's on the HN frontpage right now > this thing is clearly trained via RL to think and solve tasks for specific reasoning benchmarks. not…

The strategy of Phi isn't bad, it's just not general. It's really a model that's meant to be fine tuned, but unfortunately fine tuning tends to shit on RL'd behavior, so it ended up not being that useful. If someone made a Phi style model with an architecture that was designed to take knowledge adapters/experts (i.e. small MoE model designed to get separately trained networks plugged into them with routing updates vi…

The Phi strategy is bad. It results in very bad models that are useless in production, while gaming the benchmark to appear like it is actually able to do something. This is objectively bad.

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#29

Post by adastra22 »

roscas wrote: post #10
unstatusthequo wrote: post #7

Earlier quoted context omitted.

Yes. I tried to ask oss-gpt to ask me a riddle. The response was absurd. Came up with a nonsensical question, then told me the answer. The answer was a four letter “word” that wasn’t actually a real word. “What is the word that starts with S, ends with E, and contains A? → SAEA” Then when I said that’s not a word and you gave me the answer already, no fun, it said “I do not have access to confirm that word.”

"The word is SAME." for qwen3-coder and it took like 10 seconds to write all the thinking because it writes it down, but processing was 0 seconds. "I need to find a word that starts with S, ends with E, and contains A. Let me think through this step by step: Starts with S Ends with E Contains A somewhere in the middle One word that fits this pattern is "SAGE" - but that only has 4 letters and would be S-A-G-E, which…

He was asking the llm to come up with the riddle.

Re: GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2

#30

Post by SV_BubbleTime »

roscas wrote: post #9

From my experience, qwen3-coder is way better. I only have gpt-oss:20b installed to make a few more tests but I give it a program to make a summary of what it does and qwen3 just works in a few seconds, while gpt-oss was cancelled after 5 minuts... doing nothing. So I just use qwen3. Fast and great ouput. If for some reason I don't get what I need, I might use search engines or Perplexity. I have a 10GB 3080 and Ryze…

Are you using this in an agentic way or in a copy and paste and “code this” single input single output way?

I’d like to know how far the frontier models are from the local for agentic coding.

Post reply on HN