Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI o3 and o4-mini

openai.com

Post reply on HN
Print view View original

211–220 of 527 posts

Re: OpenAI o3 and o4-mini

#211

Post by iandanforth »

o3 failed the first test I gave it. I wanted it to create a bar chart using Python of the first 10 Fibonacci numbers (did this easily), and then use that image as input to generate an info-graphic of the chart with an animal theme. It failed in two ways. It didn't have access to the visual output from python and, when I gave it a screenshot of that output, it failed in standard GenAI fashion by having poor / incomplete text and not adhering exactly to bar heights, which were critical in this case.

So one failure that could be resolved with better integration on the back end and then an open problem with image generation in general.

Re: OpenAI o3 and o4-mini

#212

Post by pixl97 »

jcynix wrote: post #80

To plan a visit to a dark sky place, I used duck.ai (Duckduckgo's experimental AI chat feature) to ask five different AIs on what date the new moon will happen in August 2025. GPT-4o mini: The new moon in August 2025 will occur on August 12. Llama 3.3 70B: The new moon in August 2025 is expected to occur on August 16, 2025. Claude 3 Haiku: The new moon in August 2025 will occur on August 23, 2025. o3-mini: Based on a…

So I asked GPT-o4-mini-high

"On what date will the new moon occur on in August 2025. Use a tool to verify the date if needed"

It correctly reasoned it did not have exact dates due to its cutoff and did a lookup.

"The new moon in August 2025 falls on Friday, August 22, 2025"

Now, I did not specify the timezone I was in so our timing between 22 and 23 appears to be just a time zone difference at it had marked an time of 23:06 PDT per its source.

Re: OpenAI o3 and o4-mini

#213

Post by erikw »

Interesting... I asked o3 for help writing a flake so I could install the latest Webstorm on NixOS (since the one in the package repo is several months old), and it looks like it actually spun up a NixOS VM, downloaded the Webstorm package, wrote the Flake, calculated the SHA hash that NixOS needs, and wrote a test suite. The test suite indicates that it even did GUI testing- not sure whether that is a hallucination or not though. Nevertheless, it one-shotted the installation instructions for me, and I don't see how it could have calculated the package hash without downloading, so I think this indicates some very interesting new capabilities. Highly impressive.

Re: OpenAI o3 and o4-mini

#214

Post by sksxihve »

w10-1 wrote: post #203
_fat_santa wrote: post #58

So at this point OpenAI has 6 reasoning models, 4 flagship chat models, and 7 cost optimized models. So that's 17 models in total and that's not even counting their older models and more specialized ones. Compare this with Anthropic that has 7 models in total and 2 main ones that they promote. This is just getting to be a bit much, seems like they are trying to cover for the fact that they haven't actually done much.…

> This is just getting to be a bit much, seems like they are > trying to cover for the fact that they haven't actually done much Or perhaps they're trying to make some important customers happy by showing movement on areas the customers care about. Subjectively, customers get locked in by feeling they have the inside track, and these small tweaks prove that. Objectively, the small change might make a real difference…

> Or perhaps they're trying to make some important customers happy by showing movement on areas the customers care about

Or make important investors happy, they need to justify the latest $40 billion round

Re: OpenAI o3 and o4-mini

#215

Post by anothermathbozo »

carlita_express wrote: post #77

> we’ve observed that large-scale reinforcement learning exhibits the same “more compute = better performance” trend observed in GPT‑series pretraining. Didn’t the pivot to RL from pretraining happen because the scaling “law” didn’t deliver the expected gains? (Or at least because O(log) increases in model performance became unreasonably costly?) I see they’ve finally resigned themselves to calling these trends, not…

This isn't exactly the case. The trend is a log scale. So a 10x in pretraining should yield a 10% increase in performance. That's not proving to be false per say but rather they are encountering practical limitations around 10x'ing data volume and 10x'ing available compute.

Re: OpenAI o3 and o4-mini

#216

Post by littlestymaar »

Workaccount2 wrote: post #111
emp17344 wrote: post #37

Earlier quoted context omitted.

Not really. We’re definitely in the incremental improvement stage at this point. Certainly no indication that progress is “accelerating”.

Integration is accelerating rapidly. Even if model development froze today, we would still probably have ~5 years of adoption and integration before it started to level off.

You are both correct. It feels like the tech itself is kinda plateauing but it's still massively under-used. It will take a decade or more before the deployment starts slowing down.

Re: OpenAI o3 and o4-mini

#217

Post by pizzathyme »

georgewsinger wrote: post #32

Very impressive! But under arguably the most important benchmark -- SWE-bench verified for real-world coding tasks -- Claude 3.7 still remains the champion.[1] Incredible how resilient Claude models have been for best-in-coding class. [1] But by only about 1%, and inclusive of Claude's "custom scaffold" augmentation (which in practice I assume almost no one uses?). The new OpenAI models might still be effectively bes…

The image generation improvement with o4-mini is incredible. Testing it out today, this is a step change in editing specificity even from the ChatGPT 4o LLM image integration just a few weeks ago (which was already a step change). I'm able to ask for surgical edits, and they are done correctly.

There isn't a numerical benchmark for this that people seem to be tracking but this opens up production-ready image use cases. This was worth a new release.

Re: OpenAI o3 and o4-mini

#218

Post by erikw »

unsupp0rted wrote: post #85
jjani wrote: post #63

Earlier quoted context omitted.

Gemini 2.5 Pro is widely considered superior to 3.7 Sonnet now by heavy users, but they don't have an SWE-bench score. Shows that looking at one such benchmark isn't very telling. Main advantage over Sonnet being that it's better at using a large amount of context, which is enormously helpful during coding tasks. Sonnet is still an incredibly impressive model as it held the crown for 6 months, which may as well be a…

Main advantage over Sonnet is Gemini 2.5 doesn't try to make a bunch of unrelated changes like it's rewriting my project from scratch.

What language / framework are you using? I ask because in a Node / Typescript / React project I experience the opposite- Claude 3.7 usually solves my query on the first try, and seems to understand the project's context, ie the file structure, packages, coding guidelines, tests, etc, while Gemini 2.5 seems to install packages willy-nilly, duplicate existing tests, create duplicate components, etc.

Re: OpenAI o3 and o4-mini

#219

Post by shmatt »

pants2 wrote: post #152
shmatt wrote: post #120

Earlier quoted context omitted.

Im old enough to remember the mystery and hype before o*/o1/strawberry that was supposed to be essentially AGI. We had serious news outlets write about senior people at OpenAI quitting because o1 was SkyNet Now we're up to o4, AGI is still not even in near site (depending on your definition, I know). And OpenAI is up to about 5000 employees. I'd think even before AGI a new model would be able to cover for at least 45…

Remember that Docusign has 7,000 employees. I think OpenAI is pretty lean for what they're accomplishing.

If we're making comparisons, its more like someone selling a $10,000 course on how to be a millionaire

Not directly from OpenAI - but people in the industry is advertising how these advanced models can replace employees, yet they keep on going on hiring tears (including OpenAI). Lets see the first company to stand behind their models, and replace 50% of their existing headcount with agents. That to me would be a sign these things are going to replace peoples jobs. Until I see that, if OpenAI can't figure out how to replace humans with models, then no one will

I mean could you imagine if todays announcement was - the chatgpt.com webdev team has been laid off, and all new features and fixes will be complete by Codex CLI + o4-mini. That means they believe in the product theyre advertising. Until they do something like that, theyll keep on trusting those human engineers and try selling other people on the dream

Re: OpenAI o3 and o4-mini

#220

Post by vunderba »

_fat_santa wrote: post #58

So at this point OpenAI has 6 reasoning models, 4 flagship chat models, and 7 cost optimized models. So that's 17 models in total and that's not even counting their older models and more specialized ones. Compare this with Anthropic that has 7 models in total and 2 main ones that they promote. This is just getting to be a bit much, seems like they are trying to cover for the fact that they haven't actually done much.…

> This is just getting to be a bit much, seems like they are trying to cover for the fact that they haven't actually done much.

Did you miss the 4o image generation announcement from roughly three week ago?

https://news.ycombinator.com/item?id=43474112

Combining a multimodal LLM+ImageGen puts them pretty significantly ahead of the curve at least in that domain.

Demonstration of the capabilities:

https://mordenstar.com/blog/chatgpt-4o-images

Post reply on HN