Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI o3 and o4-mini

openai.com

Post reply on HN
Print view View original

131–140 of 527 posts

Re: OpenAI o3 and o4-mini

#131

Post by tedsanders »

georgewsinger wrote: post #94
oofbaroomf wrote: post #53

Earlier quoted context omitted.

Claude got 63.2% according to the swebench.com leaderboard (listed as "Tools + Claude 3.7 Sonnet (2025-02-24)).[0] OpenAI said they got 69.1% in their blog post. [0] swebench.com/#verified

Yes, however Claude advertised 70.3%[1] on SWE bench verified when using the following scaffolding: > For Claude 3.7 Sonnet and Claude 3.5 Sonnet (new), we use a much simpler approach with minimal scaffolding, where the model decides which commands to run and files to edit in a single session. Our main “no extended thinking” pass@1 result simply equips the model with the two tools described here—a bash tool, and a fi…

I think you may have misread the footnote. That simpler setup results in the 62.3%/63.7% score. The 70.3% score results from a high-compute parallel setup with rejection sampling and ranking:

> For our “high compute” number we adopt additional complexity and parallel test-time compute as follows:

> We sample multiple parallel attempts with the scaffold above

> We discard patches that break the visible regression tests in the repository, similar to the rejection sampling approach adopted by Agentless; note no hidden test information is used.

> We then rank the remaining attempts with a scoring model similar to our results on GPQA and AIME described in our research post and choose the best one for the submission.

> This results in a score of 70.3% on the subset of n=489 verified tasks which work on our infrastructure. Without this scaffold, Claude 3.7 Sonnet achieves 63.7% on SWE-bench Verified using this same subset.

Re: OpenAI o3 and o4-mini

#132

Post by Topfi »

I have barely found time to gauge 4.1s capabilities, so at this stage, I’d rather focus on the ever worsening names these companies bestow upon their models. To say that I the USB-IF have found their match would be an understatement.

Re: OpenAI o3 and o4-mini

#133

Post by qoez »

jdross wrote: post #6

The pace of notable releases across the industry right now is unlike any time I remember since I started doing this in the early 2000's. And it feels like it's accelerating

Lots of releases but very little actual performance increases

Re: OpenAI o3 and o4-mini

#135

Post by zaptrem »

unsupp0rted wrote: post #85
jjani wrote: post #63

Earlier quoted context omitted.

Gemini 2.5 Pro is widely considered superior to 3.7 Sonnet now by heavy users, but they don't have an SWE-bench score. Shows that looking at one such benchmark isn't very telling. Main advantage over Sonnet being that it's better at using a large amount of context, which is enormously helpful during coding tasks. Sonnet is still an incredibly impressive model as it held the crown for 6 months, which may as well be a…

Main advantage over Sonnet is Gemini 2.5 doesn't try to make a bunch of unrelated changes like it's rewriting my project from scratch.

I do find it likes to subtly reformat every single line thereby nuking my diff and making its changes unusable since I can’t verify them that way, which Sonnet doesn’t do.

Re: OpenAI o3 and o4-mini

#136

Post by pembrook »

evaneykelen wrote: post #49

A suggestion for OpenAI to create more meaningful model names: {Size}-{Quarter/Year}-{Speed/Accuracy}-{Specialty} Where: * Size is XS/S/M/L/XL/XXL to indicate overall capability level * Quarter/Year like Q2-25 * Speed/Accuracy indicated as Fast/Balanced/Precise * Optional specialty tag like Code/Vision/Science/etc Example model names: * L-Q2-25-Fast-Code (Large model from Q2 2025, optimized for speed, specializes in…

Thank god we don’t usually let engineers name stuff in the west.

While this is entirely logical in theory this is how you get LG style naming like “THE ALL NEW LG-CFT563-X2”

I mean, it makes total sense, it tells you exactly the model, region, series and edition! Right??

Re: OpenAI o3 and o4-mini

#137

Post by whalesalad »

_fat_santa wrote: post #58

So at this point OpenAI has 6 reasoning models, 4 flagship chat models, and 7 cost optimized models. So that's 17 models in total and that's not even counting their older models and more specialized ones. Compare this with Anthropic that has 7 models in total and 2 main ones that they promote. This is just getting to be a bit much, seems like they are trying to cover for the fact that they haven't actually done much.…

Model fatigue is a real thing - Particularly with their billing model that is wildly different from model to model and gives you more headroom as you spend more. We spend a lot of time and effort running tests across many models to balance for that cost/performance ratio. When you can run 300k tokens per min on a shittier model, or 10k tokens per min on a better model - you want to use the cheaper model but if the performance isn't there then you gotta pivot. Can I use tools here? Can I use function calling here? Do I use the chat API, the chat completions API, or the responses API? Do either of those work with the model I want to use, or only with other models?

I almost wonder if this is intentional ... because when you create a quagmire of insane inter-dependent billing scenarios you end up with a product like AWS that can generate substantial amounts of revenue from sheer ignorance or confusion. Then you can hire special consultants to come in and offer solutions to your customers in order to wade through the muck on your behalf.

Dealing with OpenAI's API's is a straight up nightmare.

Re: OpenAI o3 and o4-mini

#138

Post by SweetSoftPillow »

brap wrote: post #4

Where's the comparison with Gemini 2.5 Pro?

Some sources mention that o3 scores 63.8 on SWE-bench, while Gemini 2.5 Pro scores 69.1.

On most other benchmarks, they seem to perform about the same, which is bad news for o3 because it's much more expensive and slower than Gemini 2.5 Pro, and it also hides its reasoning while Gemini shows everything.

We can probably just stick with Gemini 2.5 Pro, since it offers the best combination of price, quality, and speed. No need to worry about finding a replacement (for now).

Re: OpenAI o3 and o4-mini

#139

Post by enraged_camel »

itsmevictor wrote: post #98
unsupp0rted wrote: post #85

Earlier quoted context omitted.

Main advantage over Sonnet is Gemini 2.5 doesn't try to make a bunch of unrelated changes like it's rewriting my project from scratch.

I find Gemini 2.5 truly remarkable and overall better than Claude, which I was a big fan of

Still doesn't work well in Cursor unfortunately.

Re: OpenAI o3 and o4-mini

#140

Post by mkozlows »

drcongo wrote: post #121
kristofferR wrote: post #79

Earlier quoted context omitted.

To use that criticism for this release ain't really fair, as these will replace the old models (o3 will replace o1, o4-mini will replace o3-mini). On a more general level - sure, but they aren't planning to use this release to add a larger number of models, it's just that deprecating/killing the old models can't be done overnight.

As someone who doesn't use anything OpenAI (for all the reasons), I have to agree with the GP. It's all baffling. Why is there an o3-mini and an o4-mini? Why on earth are there so many models? Once you get to this point you're putting the paradox of choice on the user - I used to use a particular brand toothpaste for years until it got to the point where I'd be in the supermarket looking at a wall of toothpaste all b…

They keep a lot of models around for backward compatibility for API users. This is confusing, but not inherently a bad idea.
Post reply on HN