Open models by OpenAI

Post by __alexs » Wed, Aug 06, 2025, 8:58 AM UTC

dragonwriter wrote: post #672

int_19h wrote: post #496

Earlier quoted context omitted.

I tried 20b locally and it couldn't reason a way out of a basic river crossing puzzle with labels changed. That is not anywhere near SOTA. In fact it's worse than many local models that can do it, including e.g. QwQ-32b.

> In fact it's worse than many local models that can do it, including e.g. QwQ-32b. I'm not going to be surprised that a 20B 4/32 MoE model (3.6B parameters activated) is less capable at a particular problem category than a 32B dense model, and its quite possible for both to be SOTA, as state of the art at different scale (both parameter count and speed which scales with active resource needs) is going to have differ…

Post by Bombthecat » Wed, Aug 06, 2025, 9:04 AM UTC

LinXitoW wrote: post #687

M4R5H4LL wrote: post #437

Earlier quoted context omitted.

+1 - I work in finance, and there's no way we're sending our data and code outside the organization. We have our own H100s.

Possibly stupid question, but does this apply to things like M365 too? Because just like with Inference providers, the only thing keeping them from reading/abusing your data is a pinky promise contract. Basically, isn't your data as safe/unsafe in a sharepoint folder as it is sending it to a paid inference provider?

Post by jari_mustonen » Wed, Aug 06, 2025, 9:09 AM UTC

freehorse wrote: post #666

9rx wrote: post #637

Earlier quoted context omitted.

I tried the two US presidents having the same parents one, and while it understood the intent, it got caught up in being adamant that Joe Biden won the election in 2024 and anything I do to try and tell it otherwise is dismissed as being false and expresses quite definitely that I need to do proper research with legitimate sources.

I mean I would hardly blame the specific model, Anthropic has a specific mention in their system prompts on trump winning. For some reason llms get confused with this one.

Post by WmWsjA6B29B4nfk » Wed, Aug 06, 2025, 9:15 AM UTC

bigmanhank wrote: post #690

Nomadeon wrote: post #458

Earlier quoted context omitted.

Agree. Concrete example: "What was the Japanese codeword for Midway Island in WWII?" Answer on Wikipedia: https://en.wikipedia.org/wiki/Battle_of_Midway#U.S._code-bre... dolphin3.0-llama3.1-8b Q4_K_S [4.69 GB on disk]: correct in deepseek-r1-0528-qwen3-8b Q6_K [6.73 GB]: correct in 10 seconds gpt-oss-20b MXFP4 [12.11 GB] low reasoning: wrong after 6 seconds gpt-oss-20b MXFP4 [12.11 GB] high reasoning: wrong after 3 m…

Not true: During World War II the Imperial Japanese Navy referred to Midway Island in their communications as “Milano” (ミラノ). This was the official code word used when planning and executing operations against the island, including the Battle of Midway. 12.82 tok/sec 140 tokens 7.91s to first token openai/gpt-oss-20b

Post by CMay » Wed, Aug 06, 2025, 9:23 AM UTC

int_19h wrote: post #496

cco wrote: post #357

The lede is being missed imo. gpt-oss:20b is a top ten model (on MMLU (right behind Gemini-2.5-Pro) and I just ran it locally on my Macbook Air M3 from last year. I've been experimenting with a lot of local models, both on my laptop and on my phone (Pixel 9 Pro), and I figured we'd be here in a year or two. But no, we're here today. A basically frontier model, running for the cost of electricity (free with a rounding…

I tried 20b locally and it couldn't reason a way out of a basic river crossing puzzle with labels changed. That is not anywhere near SOTA. In fact it's worse than many local models that can do it, including e.g. QwQ-32b.