Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open models by OpenAI

openai.com

Post reply on HN
Print view View original

621–630 of 909 posts

Re: Open models by OpenAI

#621

Post by cco »

MattSayar wrote: post #368
cco wrote: post #357

The lede is being missed imo. gpt-oss:20b is a top ten model (on MMLU (right behind Gemini-2.5-Pro) and I just ran it locally on my Macbook Air M3 from last year. I've been experimenting with a lot of local models, both on my laptop and on my phone (Pixel 9 Pro), and I figured we'd be here in a year or two. But no, we're here today. A basically frontier model, running for the cost of electricity (free with a rounding…

What's your experience with the quality of LLMs running on your phone?

As other said, around gpt 3.5 level so three or four years behind SOTA today at reasonable (but not quick) speed.

Re: Open models by OpenAI

#622

Post by snthpy »

cco wrote: post #357

The lede is being missed imo. gpt-oss:20b is a top ten model (on MMLU (right behind Gemini-2.5-Pro) and I just ran it locally on my Macbook Air M3 from last year. I've been experimenting with a lot of local models, both on my laptop and on my phone (Pixel 9 Pro), and I figured we'd be here in a year or two. But no, we're here today. A basically frontier model, running for the cost of electricity (free with a rounding…

For me the biggest benefit of open weights models is the ability to fine tune and adapt to different tasks.

Re: Open models by OpenAI

#623

Post by RandyOrion »

Super shallow (24/36 layers) MoE with low active parameter counts (3.6B/5.1B), a tradeoff between inference speed and performance.

Text only, which is okay.

Weights partially in MXFP4, but no cuda kernel support for RTX 50 series (sm120). Why? This is a NO for me.

Safety alignment shifts from off the charts to off the rails really fast if you keep prompting. This is a NO for me.

In summary, a solid NO for me.

Re: Open models by OpenAI

#624

Post by markasoftware »

simonw wrote: post #619
markasoftware wrote: post #589

Earlier quoted context omitted.

The space invaders game seems like a poor benchmark. Both models understood the prompt and generated valid, functional javascript. One just added more fancy graphics. It might just have "use fancy graphics" in its system prompt for all we know.

The way I run these prompts excludes a system prompt - I'm hitting the models directly.

still, if you ask this open model to generate a fancy space invaders game with polish, and then ask the other model to generate a bare-bones space invaders game with the fewest lines of code, I think there's a good chance they'd switch places. This doesn't really test the models ability to generate a space invaders game, so much as it tests their tendency to make an elaborate vs simple solution.

Re: Open models by OpenAI

#625

Post by pxc »

mudkipdev wrote: post #596
simonw wrote: post #338

Just posted my initial impressions, took a couple of hours to write them up because there's a lot in this release! https://simonwillison.net/2025/Aug/5/gpt-oss/ TLDR: I think OpenAI may have taken the medal for best available open weight model back from the Chinese AI labs. Will be interesting to see if independent benchmarks resolve in that direction as well. The 20B model runs on my Mac laptop using less than 15GB…

Hasn't nailed the strawberry test yet

I found this surprising because that's such an old test that it must certainly be in the training data. I just tried to reproduce and I've been unable to get it (20B model, lowest "reasoning" budget) to fail that test (with a few different words).

Re: Open models by OpenAI

#626

Post by Foobar8568 »

M4R5H4LL wrote: post #437
wizee wrote: post #417

Earlier quoted context omitted.

Privacy, both personal and for corporate data protection is a major reason. Unlimited usage, allowing offline use, supporting open source, not worrying about a good model being taken down/discontinued or changed, and the freedom to use uncensored models or model fine tunes are other benefits (though this OpenAI model is super-censored - “safe”). I don’t have much experience with local vision models, but for text ques…

+1 - I work in finance, and there's no way we're sending our data and code outside the organization. We have our own H100s.

Look at (private) banks in Switzerland, there are enough press release, and I can confirm most of them.

Managing private clients direct data is still a concern if it can be directly linked to them.

Only JB I believe have on premise infrastructure for these use cases.

Re: Open models by OpenAI

#627

Post by rasmus1610 »

atlasunshrugged wrote: post #573
filoleg wrote: post #549

Earlier quoted context omitted.

Add big law to the list as well. There are at least a few firms here that I am just personally aware of running their models locally. In reality, I bet there are way more.

Add government here too (along with all the firms that service government customers)

Add healthcare. Cannot send our patients data to a cloud provider

Re: Open models by OpenAI

#628

Post by h4ny »

simonw wrote: post #617
h4ny wrote: post #610

Earlier quoted context omitted.

> TLDR: I think OpenAI may have taken the medal for best available open weight model back from the Chinese AI labs. That's just straight up not the case. Not sure how you can jump to that conclusion not least when you stated that you haven't tested tool calling in your post too. Many people in the community are finding it substantially lobotomized to the point that there are "safe" memes everywhere now. Maybe you nee…

If you read my full post, it ends with this: > I’m waiting for the dust to settle and the independent benchmarks (that are more credible than my ridiculous pelicans) to roll out, but I think it’s likely that OpenAI now offer the best available open weights models. You told me off for jumping to conclusions and in the same comment quoted me saying "I think OpenAI may have taken" - that's not a conclusion, it's tentati…

I did read that and it doesn't change what I said about your comment on HN, I was calling out the fact that you are making a very bold statement without having done careful analysis.

You know you have a significant audience, so don't act like you don't know what you're doing when you chose to say "TLDR: I think OpenAI may have taken the medal for best available open weight model back from the Chinese AI labs" then defend what I was calling out based on word choices like "conclusions" (I'm sure you have read conclusions in academic journals?), "I think", and "speculation".

Re: Open models by OpenAI

#629

Post by simonw »

markasoftware wrote: post #624
simonw wrote: post #619

Earlier quoted context omitted.

The way I run these prompts excludes a system prompt - I'm hitting the models directly.

still, if you ask this open model to generate a fancy space invaders game with polish, and then ask the other model to generate a bare-bones space invaders game with the fewest lines of code, I think there's a good chance they'd switch places. This doesn't really test the models ability to generate a space invaders game, so much as it tests their tendency to make an elaborate vs simple solution.

My main goal with that benchmark is to see if it can produce HTML and JavaScript code that runs without errors for a moderately complex challenge.

It's not a comprehensive benchmark - there are many ways you could run it in ways that would be much more informative and robust.

It's great as a quick single sentence prompt to get a feeling for if the model can produce working JavaScript or not.

Re: Open models by OpenAI

#630

Post by simonw »

h4ny wrote: post #628
simonw wrote: post #617

Earlier quoted context omitted.

If you read my full post, it ends with this: > I’m waiting for the dust to settle and the independent benchmarks (that are more credible than my ridiculous pelicans) to roll out, but I think it’s likely that OpenAI now offer the best available open weights models. You told me off for jumping to conclusions and in the same comment quoted me saying "I think OpenAI may have taken" - that's not a conclusion, it's tentati…

I did read that and it doesn't change what I said about your comment on HN, I was calling out the fact that you are making a very bold statement without having done careful analysis. You know you have a significant audience, so don't act like you don't know what you're doing when you chose to say "TLDR: I think OpenAI may have taken the medal for best available open weight model back from the Chinese AI labs" then de…

I'm going to double down on "I think OpenAI may have taken the medal..." not being a "bold statement".

I try to be careful about my choice of words, even in forum comments.

Post reply on HN