Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI O3-Mini

openai.com

Post reply on HN
Print view View original

411–420 of 944 posts

Re: OpenAI O3-Mini

#411

Post by thegeomaster »

ALittleLight wrote: post #52
sss111 wrote: post #3

So far, it seems like this is the hierarchy o1 > GPT-4o > o3-mini > o1-mini > GPT-4o-mini o3 mini system card: https://cdn.openai.com/o3-mini-system-card.pdf

That seems very bad. What's the point of a new model that's worse than 4o? I guess it's cheaper in the API and a bit better at coding - but, this doesn't seem compelling. With DeepSeek I heard OpenAI saying the plan was to move releases on models that were meaningfully better than the competition. Seems like what we're getting is the scheduled releases that are worse than the current versions.

It's quite a bit better than coding --- they hint that it can tie o1's performance for coding, which already benchmarks higher than 4o. And it's significantly cheaper, and presumably faster. I believe API costs account for the vast majority of COGS at most today's AI startups, so they would be very motivated to switch to a cheaper model that has similar performance.

Re: OpenAI O3-Mini

#412

Post by cm2187 »

dkjaudyeqooe wrote: post #270
jen729w wrote: post #211

> Testers preferred o3-mini's responses to o1-mini 56% of the time I hope by this they don't mean me, when I'm asked 'which of these two responses do you prefer'. They're both 2,000 words, and I asked a question because I have something to do. I'm not reading them both ; I'm usually just selecting the one that answered first. That prompt is pointless. Perhaps as evidenced by the essentially 50% response rate: it's a…

It's kind of strange that they gave that stat. Maybe they thought people would somehow think about "56% better" or something. Because when you think about it, it really is quite damning. Minus statistical noise it's no better.

And another way to rephrase it is that almost half of the users prefer the older model, which is terrible PR.

Re: OpenAI O3-Mini

#413

Post by ern »

I haven’t bothered with o3 mini, because who wants an “inferior” product? I was using 4o as a “smarter Google” until DeepSeek appeared (although its web search is being hammered now and I’m just using Google ).

o1 seems to have been neutered in the last week lots of disclaimers and butt-covering in its responses.

I also had an annoying discussion with o1 about the DC plane crash..it doesn’t have web access and its cutoff is 2024, so I don’t expect it know about the crash. However, after saying such an event is extremely unlikely and being almost patronisingly reassuring, it treated pasted news articles and links (which to be sure, it can’t access) as “fictionalized”, instead of acknowledging its own cut-off date, and that it could have been wrong. In contrast DeepSeek (with web search turned off) was less dismissive of the risks in DC airspace, and more aware of its own knowledge cut-off.

Coupled with the limited number of o1 responses for ChatGPT Plus, I’ve cancelled my subscription for now.

Re: OpenAI O3-Mini

#414

Post by mediaman »

tucnak wrote: post #381
msp26 wrote: post #60

I wish they'd just reveal the CoT (like gemini and deepseek do), it's very helpful to see when the model gets misled by something in your prompt. Paying for tokens you aren't even allowed to see is peak OpenAI.

I'm sorry, but it's over for OpenAI. Some have predicted this; including me back in November[1] when I wrote "o1 is a revolution in accounting, not capability" which although tongue-in-cheek, has so far turned out to be correct. I'm only waiting to see what Google, Facebook et al. will accomplish now that R1-Zero result is out the bag. The nerve, the cheek of this hysterical o3-mini release—insisting to hide the COT…

You don't need to wait for Google. Their Jan 21 checkpoint for their fast reasoning model is available on AIStudio. It shows full reasoning traces. It's very good, much faster than R1, and although they haven't released pricing, based on flash it's going to be quite cheap.

Re: OpenAI O3-Mini

#415

Post by BeetleB »

scarface_74 wrote: post #314
BeetleB wrote: post #294

Earlier quoted context omitted.

I would not expect any LLM to get this right. I think people have too high expectations for it. Now if you asked it to write a Python program to list them in order, and have it enter all the names, birthdays, and year elected in a list to get the program to run - that's more reasonable.

The “o” models get the order right. DeepSeek also gets the order right. It doesn’t show on the share link. But it actually outputs the list correctly from the built in Python interpreter. For some things, ChatGPT 4o will automatically use its Python runtime

That some models get it right is irrelevant. In general, if your instructions require computation, it's safer to assume it won't get it right and will hallucinate.

Re: OpenAI O3-Mini

#416

Post by n2d4 »

fkyoureadthedoc wrote: post #78
n2d4 wrote: post #64

Earlier quoted context omitted.

> They really need someone in marketing. Who said this is not intentional? It seems to work well given that people are hyped every time there's a release, no matter how big the actual improvements are — I'm pretty sure "o3-mini" works better for that purpose than "GPT 4.1.3"

> I'm pretty sure "o3-mini" works better for that purpose than "GPT 4.1.3" Why would the marketing team of all people call it GPT 4.1.3?

They wouldn't! They would call it o3-mini, even though GPT 4.1.3 may or may not "make more sense" from a technical perspective.

Re: OpenAI O3-Mini

#417

Post by n2d4 »

throwaway287391 wrote: post #409
n2d4 wrote: post #405

Earlier quoted context omitted.

Because you're not testing "will a user click the left or right button" (for which asking a thousand users to click a button would be a pretty good estimation), you're testing "which response is preferred". If 10% of people just click based on how fast the response was because they don't want to read both outputs, your p-value for the latter hypothesis will be atrocious, no matter how large the sample is.

Yes, I am assuming they evaluated the models in good faith, understand how to design a basic user study, and therefore when they ran a study intended to compare the response quality between two different models, they showed the raters both fully-formed responses at the same time, regardless of the actual latency of each model.

I would recommend you read the comment that started this thread then, because that's the context we're talking about: https://news.ycombinator.com/item?id=42891294
Post reply on HN