Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open models by OpenAI

openai.com

Post reply on HN
Print view View original

751–760 of 909 posts

Re: Open models by OpenAI

#751

Post by diggan »

freehorse wrote: post #666
9rx wrote: post #637

Earlier quoted context omitted.

I tried the two US presidents having the same parents one, and while it understood the intent, it got caught up in being adamant that Joe Biden won the election in 2024 and anything I do to try and tell it otherwise is dismissed as being false and expresses quite definitely that I need to do proper research with legitimate sources.

I mean I would hardly blame the specific model, Anthropic has a specific mention in their system prompts on trump winning. For some reason llms get confused with this one.

I noticed the same when asking various LLMs to summarize and explaining some "Presidential Actions" (from https://www.whitehouse.gov/presidential-actions/), most of them answer "This is just theoretical, since no such executive actions actually exists, but assuming something like that would happen in the future, it would mean ..." while a few has returned something like "This fictional executive action would be illegal so I cannot summarize the content", even when I provide direct links and they fetch the content themselves. Not exactly sure why that is.

Re: Open models by OpenAI

#752

Post by seba_dos1 »

Nomadeon wrote: post #458
dongobread wrote: post #433

Earlier quoted context omitted.

How up to date are you on current open weights models? After playing around with it for a few hours I find it to be nowhere near as good as Qwen3-30B-A3B. The world knowledge is severely lacking in particular.

Agree. Concrete example: "What was the Japanese codeword for Midway Island in WWII?" Answer on Wikipedia: https://en.wikipedia.org/wiki/Battle_of_Midway#U.S._code-bre... dolphin3.0-llama3.1-8b Q4_K_S [4.69 GB on disk]: correct in deepseek-r1-0528-qwen3-8b Q6_K [6.73 GB]: correct in 10 seconds gpt-oss-20b MXFP4 [12.11 GB] low reasoning: wrong after 6 seconds gpt-oss-20b MXFP4 [12.11 GB] high reasoning: wrong after 3 m…

How would asking this kind of question without providing the model with access to Wikipedia be a valid benchmark for anything useful?

Re: Open models by OpenAI

#754

Post by FrustratedMonky »

cpursley wrote: post #733
ben_w wrote: post #727

Earlier quoted context omitted.

It's also extremely weird that Trump did win in 2024. If I'd been in a coma from Jan 1 2024 to today, and woke up to people saying Trump was president again, I'd think they were pulling my leg or testing my brain function to see if I'd become gullible.

You’re in a bubble. It was no surprise to folks who touch grass on the regular.

[flagged]

Re: Open models by OpenAI

#755

Post by dennisy »

simonw wrote: post #629
markasoftware wrote: post #624

Earlier quoted context omitted.

still, if you ask this open model to generate a fancy space invaders game with polish, and then ask the other model to generate a bare-bones space invaders game with the fewest lines of code, I think there's a good chance they'd switch places. This doesn't really test the models ability to generate a space invaders game, so much as it tests their tendency to make an elaborate vs simple solution.

My main goal with that benchmark is to see if it can produce HTML and JavaScript code that runs without errors for a moderately complex challenge. It's not a comprehensive benchmark - there are many ways you could run it in ways that would be much more informative and robust. It's great as a quick single sentence prompt to get a feeling for if the model can produce working JavaScript or not.

Not really the other commenters are correct I feel and this is not really proving anything about the fundamental capability of the model. It’s just a hello world benchmark adding no real value, just driving blog traffic for you.

Re: Open models by OpenAI

#756

Post by bavell »

simonw wrote: post #630
h4ny wrote: post #628

Earlier quoted context omitted.

I did read that and it doesn't change what I said about your comment on HN, I was calling out the fact that you are making a very bold statement without having done careful analysis. You know you have a significant audience, so don't act like you don't know what you're doing when you chose to say "TLDR: I think OpenAI may have taken the medal for best available open weight model back from the Chinese AI labs" then de…

I'm going to double down on "I think OpenAI may have taken the medal..." not being a "bold statement". I try to be careful about my choice of words, even in forum comments.

> I think OpenAI may have taken the medal for best available open weight model back from the Chinese AI labs.

IMO, the "I think..." bit could be ambiguous and read as, "In my opinion, OpenAI may have...".

I agree with you it's not a hard/bold endorsement but perhaps leading with the disclaimer that you're reserving final judgement could assuage these concerns.

Re: Open models by OpenAI

#757

Post by PeterStuer »

phh wrote: post #373
PeterStuer wrote: post #117

I love how they frame High-end desktops and laptops as having "a single H100 GPU".

Well if nVidia wasn't late, it would be runnable on nVidia project Digits.

Yes, they are late to the party. Maybe they do not want to eat into the RTX Pro 6000 sales. In the meantime, there is the AMD Ryzen™ Al Max+ 395.

Re: Open models by OpenAI

#758

Post by ricardobayes »

captainregex wrote: post #397
cco wrote: post #357

The lede is being missed imo. gpt-oss:20b is a top ten model (on MMLU (right behind Gemini-2.5-Pro) and I just ran it locally on my Macbook Air M3 from last year. I've been experimenting with a lot of local models, both on my laptop and on my phone (Pixel 9 Pro), and I figured we'd be here in a year or two. But no, we're here today. A basically frontier model, running for the cost of electricity (free with a rounding…

I’m still trying to understand what is the biggest group of people that uses local AI (or will)? Students who don’t want to pay but somehow have the hardware? Devs who are price conscious and want free agentic coding? Local, in my experience, can’t even pull data from an image without hallucinating (Qwen 2.5 VI in that example). Hopefully local/small models keep getting better and devices get better at running bigger…

I would say, any company who doesn't have their own AI developed. You always hear companies "mandating" AI usage, but for the most part it's companies developing their own solutions/agents. No self-respecting company with a tight opsec would allow a random "always-online" LLM that could just rip your codebase either piece by piece or the whole thing at once if it's a IDE addon (or at least I hope that's the case). So yeah, I'd say locally deployed LLM's/Agents are a gamechanger.

Re: Open models by OpenAI

#759

Post by mythz »

mythz wrote: post #168

Getting great performance running gpt-oss on 3x A4000's: gpt-oss:20b = ~46 tok/s More than 2x faster than my previous leading OSS models: mistral-small3.2:24b = ~22 tok/s gemma3:27b = ~19.5 tok/s Strangely getting nearly the opposite performance running on 1x 5070 Ti: mistral-small3.2:24b = ~39 tok/s gpt-oss:20b = ~21 tok/s Where gpt-oss is nearly 2x slow vs mistral-small 3.2.

ok issue is with ollama as gpt-oss 20B runs much faster on 1x 5070 Ti with llama.cpp and LM Studio:

    llama-server     = ~181 tok/s
    LM Studio        = ~46 tok/s  (default)
    LM Studio Custom = ~158 tok/s (changed to offload to GPU and switch to CUDA llama.cpp engine)
and llama-server on my 3x A4000 GPU Server is getting 90 tok/s vs 46 tok/s on ollama

Re: Open models by OpenAI

#760

Post by megaloblasto »

Bombthecat wrote: post #693
LinXitoW wrote: post #687

Earlier quoted context omitted.

Possibly stupid question, but does this apply to things like M365 too? Because just like with Inference providers, the only thing keeping them from reading/abusing your data is a pinky promise contract. Basically, isn't your data as safe/unsafe in a sharepoint folder as it is sending it to a paid inference provider?

Yap, companies are just paranoid, because it's new. Just like the cload back then. Sooner or later everyone will use an ai provider

A lot of people and companies use local storage and compute instead of the cloud. Cloud data is leaked all the time.
Post reply on HN