Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI o3 and o4-mini

openai.com

Post reply on HN
Print view View original

401–410 of 527 posts

Re: OpenAI o3 and o4-mini

#401

Post by ec109685 »

mchusma wrote: post #311
pizzathyme wrote: post #217

Earlier quoted context omitted.

The image generation improvement with o4-mini is incredible. Testing it out today, this is a step change in editing specificity even from the ChatGPT 4o LLM image integration just a few weeks ago (which was already a step change). I'm able to ask for surgical edits, and they are done correctly. There isn't a numerical benchmark for this that people seem to be tracking but this opens up production-ready image use case…

Thanks for sharing that. that was more interesting then their demo. I tried it and it was pretty good! I have felt that the ability to iterate from images blocked this from any real production use I had. This may be good enough now. Example of edits (not quite surgical but good): https://chatgpt.com/share/68001b02-9b4c-8012-a339-73525b8246...

I don’t know if they let you share the actual images when sharing a chat. For me, they are blank.

Re: OpenAI o3 and o4-mini

#402

Post by sealeck »

Havoc wrote: post #379
feelingsonice wrote: post #368

I’m having very mixed feelings about it. I’m using o3 to help me parse and understand a book about statistics and ML, it’s very dense in math. On one hand the answers became a lot more comprehensive and deep. It’s now able to give me very advanced explanations. On the other hand, it started overloading the answers with information. Entire concepts became single sentence summaries. Complex topics and theorems became a…

Pretty wild that we’re at the point that the human is the limitation

Surprise, the machine that interpolates from a database of maths books confuses a human who wants to learn about the contents of the books in that database.

Re: OpenAI o3 and o4-mini

#403

Post by ec109685 »

pixl97 wrote: post #212
jcynix wrote: post #80

To plan a visit to a dark sky place, I used duck.ai (Duckduckgo's experimental AI chat feature) to ask five different AIs on what date the new moon will happen in August 2025. GPT-4o mini: The new moon in August 2025 will occur on August 12. Llama 3.3 70B: The new moon in August 2025 is expected to occur on August 16, 2025. Claude 3 Haiku: The new moon in August 2025 will occur on August 23, 2025. o3-mini: Based on a…

So I asked GPT-o4-mini-high "On what date will the new moon occur on in August 2025. Use a tool to verify the date if needed" It correctly reasoned it did not have exact dates due to its cutoff and did a lookup. "The new moon in August 2025 falls on Friday, August 22, 2025" Now, I did not specify the timezone I was in so our timing between 22 and 23 appears to be just a time zone difference at it had marked an time o…

Even with a knowledge cutoff, you could know when a future new moon would be.

Re: OpenAI o3 and o4-mini

#404

Post by ec109685 »

WhatIsDukkha wrote: post #86
jcynix wrote: post #80

To plan a visit to a dark sky place, I used duck.ai (Duckduckgo's experimental AI chat feature) to ask five different AIs on what date the new moon will happen in August 2025. GPT-4o mini: The new moon in August 2025 will occur on August 12. Llama 3.3 70B: The new moon in August 2025 is expected to occur on August 16, 2025. Claude 3 Haiku: The new moon in August 2025 will occur on August 23, 2025. o3-mini: Based on a…

I would never ask any of these questions of an LLM (and I use and rely on LLMs multiple times a day), this is a job for a computer. I would also never ask a coworker for this precise number either.

These models are proclaiming near AGI, so they should be smarter than hallucinating an answer.

Re: OpenAI o3 and o4-mini

#405

Post by hirvi74 »

sharkjacobs wrote: post #385
blueprint wrote: post #210

Earlier quoted context omitted.

how do you deal with the fact that they use all of your data for training their own systems and review all conversations

gemini-2.5-pro-preview-03-25 is the paid version which doesn't use your data https://ai.google.dev/gemini-api/terms#data-use-paid

I do not feel like I can trust the empire that was built off selling personal data.

Make no mistake, I doubt the other options are trustworthy too.

Re: OpenAI o3 and o4-mini

#406

Post by ianbutler »

georgewsinger wrote: post #244
tedsanders wrote: post #131

Earlier quoted context omitted.

I think you may have misread the footnote. That simpler setup results in the 62.3%/63.7% score. The 70.3% score results from a high-compute parallel setup with rejection sampling and ranking: > For our “high compute” number we adopt additional complexity and parallel test-time compute as follows: > We sample multiple parallel attempts with the scaffold above > We discard patches that break the visible regression test…

Somehow completely missed that, thanks! I think reading this makes it even clearer that the 70.3% score should just be discarded from the benchmarks. "I got a 7%-8% higher SWE benchmark score by doing a bunch of extra work and sampling a ton of answers" is not something a typical user is going to have already set up when logging onto Claude and asking it a SWE style question. Personally, it seems like an illegitimate…

It isn't on the benchmark https://www.swebench.com/#verified

The one on the official leaderboard is the 63% score. Presumably because of all the extra work they had to do for the 70% score.

Re: OpenAI o3 and o4-mini

#407

Post by usaar333 »

SweetSoftPillow wrote: post #138
brap wrote: post #4

Where's the comparison with Gemini 2.5 Pro?

Some sources mention that o3 scores 63.8 on SWE-bench, while Gemini 2.5 Pro scores 69.1. On most other benchmarks, they seem to perform about the same, which is bad news for o3 because it's much more expensive and slower than Gemini 2.5 Pro, and it also hides its reasoning while Gemini shows everything. We can probably just stick with Gemini 2.5 Pro, since it offers the best combination of price, quality, and speed.…

> Some sources mention that o3 scores 63.8 on SWE-bench, while Gemini 2.5 Pro scores 69.1.

It's the opposite. o3 scores higher

Re: OpenAI o3 and o4-mini

#408

Post by dmd »

tedsanders wrote: post #388
dmd wrote: post #357

Earlier quoted context omitted.

Any idea when v1/models will be updated? As of right now, https://api.openai.com/v1/models has "id": "o3-mini-2025-01-31" and "id": "o3-mini", but no just 'o3'.

Ah, I know this is a pain, but by default o3 is only available to developers on tiers 4–5. If you're in tiers 1–3, you can still get access - you just need to verify your org with us here: https://help.openai.com/en/articles/10910291-api-organizatio... I recognize that verification is annoying, but we eventually had to resort to this as otherwise bad actors will create zillions of accounts to violate our policies and…

Aha! Verified and now I see o3. Thanks.

Re: OpenAI o3 and o4-mini

#409

Post by mickael-kerjean »

lattalayta wrote: post #93
georgewsinger wrote: post #32

Very impressive! But under arguably the most important benchmark -- SWE-bench verified for real-world coding tasks -- Claude 3.7 still remains the champion.[1] Incredible how resilient Claude models have been for best-in-coding class. [1] But by only about 1%, and inclusive of Claude's "custom scaffold" augmentation (which in practice I assume almost no one uses?). The new OpenAI models might still be effectively bes…

I haven't been following them that closely, but are people finding these benchmarks relevant? It seems like these companies could just tune their models to do well on particular benchmarks

The benchmark is something you can optimize for, doesn't mean it generalize well. Yesterday I tried for 2 hours to get claude to create a program that would extract data from a weird adobe file. 10$ later, the best I had is a program that was doing something like:

  switch(testFile) {
    case "test1.ase": // run this because it's a particular case 
    case "test2.ase": // run this because it's a particular case
    default:  // run something that's not working but that's ok because the previous case should
              // give the right output for all the test files ...
  }

Re: OpenAI o3 and o4-mini

#410

Post by famouswaffles »

int_19h wrote: post #393
famouswaffles wrote: post #339

Earlier quoted context omitted.

>These models cannot even make legal chess moves. That’s incredibly basic logic, and it shows how LLMs are still completely incapable of reasoning or understanding. Yeah they can. There's a link I shared to prove it which you've conveniently ignored. LLMs learn by predicting, failing and getting a little better, rinse and repeat. Pre-training is not like reading a book. LLMs trained on chess games play chess just fin…

I think the point here is that if you have to pretrain it for every specific task, it's not artificial general intelligence, by definition.

There isn't any general intelligence that isn't receiving pre-traning. People spend 14 to 18+ years in school to have any sort of career.

You don't have to pretrain it for every little thing but it should come as no surprise that a complex non-trivial game would require it.

Even if you explained all the rules of chess clearly to someone brand new to it, it will be a while and lots of practice before they internalize it.

And like I said, LLM pre-training is less like a machine reading text and more like Evolution. If you gave a corpus of chess rules, you're only training a model that knows how to converse about chess rules.

Do humans require less 'pre-training' ? Sure, but then again, that's on the back of millions of years of evolution. Modern NNs initialize random weights and have relatively very little inductive bias.

Post reply on HN