OpenAI claims gold-medal performance at IMO 2025 • Hacker News Board

Re: OpenAI claims gold-medal performance at IMO 2025

#601

Post by KoolKat23 » Sat, Jul 19, 2025, 10:40 PM UTC

SonOfLilit wrote: post #600

torginus wrote: post #573

Earlier quoted context omitted.

From my vague rememberance of doing data science years ago, it's very hard not to leak the training set. Basically how you do RL is that you make a set of training examples of input-output pairs, and set aside a smaller validation set, which you never train on, to check if your model's doing well. What you do is you tweak the architecture and the training set until it does well on the validation set. By doing so, you…

The questions were published a few days ago. The 2025 IMO just ended.

And the model was in lockdown to avoid this.

Re: OpenAI claims gold-medal performance at IMO 2025

#602

Post by SonOfLilit » Sat, Jul 19, 2025, 10:40 PM UTC

Invictus0 wrote: post #58

npinsker wrote: post #31

Earlier quoted context omitted.

The trouble is, getting an IMO gold medal is much easier (by frequency) than being the #1 Go player in the world, which was achieved by AI 10 years ago. I'm not sure it's enough to just gesture at the task; drilling down into precisely how it was achieved feels important. (Not to take away from the result, which I'm really impressed by!)

The "AI" that won Go was Monte Carlo tree search on a neural net "memory" of the outcome of millions of previous games; this is a LLM solving open ended problems. The tasks are hardly even comparable.

I really don't like the use of the word memory here, even in quotes. AlphaGo has a much better "understanding" of Go positions than mine (7k).

Re: OpenAI claims gold-medal performance at IMO 2025

#603

Post by [deleted] » Sat, Jul 19, 2025, 10:46 PM UTC

johnecheck wrote: post #48

Wow. That's an impressive result, but how did they do it? Wei references scaling up test-time compute, so I have to assume they threw a boatload of money at this. I've heard talk of running models in parallel and comparing results - if OpenAI ran this 10000 times in parallel and cherry-picked the best one, this is a lot less exciting. If this is legit, then we need to know what tools were used and how the model used…

[deleted]

Re: OpenAI claims gold-medal performance at IMO 2025

#604

Post by tim333 » Sat, Jul 19, 2025, 10:49 PM UTC

torginus wrote: post #577

modeless wrote: post #117

Noam Brown: > this isn’t an IMO-specific model. It’s a reasoning LLM that incorporates new experimental general-purpose techniques. > it’s also more efficient [than o1 or o3] with its thinking. And there’s a lot of room to push the test-time compute and efficiency further. > As fast as recent AI progress has been, I fully expect the trend to continue. Importantly, I think we’re close to AI substantially contributing…

Thing is, for example, all of classical physics can be derived from Newton's laws, Maxwell's equations and the laws of Thermodynamics, all of which can be written on a slip of paper. A sufficiently brilliant and determined human can invent or explain everything armed only with this knowledge. There's no need to train him on a huge corpus of text, like they do with ChatGPT. Not sure what this model's like, but I'm qui…

If you look at the history of physics I don't think it really worked like that. It took about three centuries from Newton to Maxwell because it's hard to just deduce everything from basic principles.

Re: OpenAI claims gold-medal performance at IMO 2025

#605

Post by gettingoverit » Sat, Jul 19, 2025, 10:57 PM UTC

tedsanders wrote: post #361

gettingoverit wrote: post #312

Earlier quoted context omitted.

Implying results are fraudulent is completely fair when it is a fraud. The previous time they had claims about solving all of the math right there and right then, they were caught owning the company that makes that independent test , and could neither admit nor deny training on closed test set.

Just to quickly clarify: - OpenAI doesn't own Epoch AI (though they did commission Epoch to make the eval) - OpenAI denied training on the test set (and further denied training on FrontierMath-derived data, training on data targeting FrontierMath specifically, or using the eval to pick a model checkpoint; in fact, they only downloaded the FrontierMath data after their o3 training set was frozen and they didn't look a…

That said, I missed the slight semantic difference between "being funded by" and "owning", even though I don't see how that would be different in practice.

Regarding the second point, I don't see how "hav[ing] a verbal agreement that these materials will not be used in model training" would actually discourage someone from not doing it, because breaking that kind of verbal agreement wouldn't cause any harm.

I have not been aware of those other claims on Twitter, but IMO they do not create sufficient basis for an investor fraud case either, because Twitter is not an official way of communcating to investors, which means they can claim whatever they want there. IANAL though.

I'm really looking for FrontierMath-level problems to be solvable by OpenAI models, and being able to validate it myself, yet I don't have much hope it will happen during my lifetime.

Re: OpenAI claims gold-medal performance at IMO 2025

#606

Post by bgwalter » Sat, Jul 19, 2025, 11:01 PM UTC

atleastoptimal wrote: post #412

>AI model performs astounding feat everyone claimed was impossible or won’t be achieved for a while >Commenters on HN claim it must not be that hard, or OpenAI is lying, or cheated. Anything but admit that it is impressive Every time on this site lol. A lot of people here have an emotional aversion to accepting AI progress. They’re deep in the bargaining/anger/denial phase.

Thank you for this pop psychology evaluation. It could have been written by an an "AI".

Re: OpenAI claims gold-medal performance at IMO 2025

#607

Post by CamperBob2 » Sat, Jul 19, 2025, 11:02 PM UTC

suddenlybananas wrote: post #592

lexandstuff wrote: post #553

Earlier quoted context omitted.

I don't fault you for maintaining a healthy scepticism, but per the President of the IMO: "It is very exciting to see progress in the mathematical capabilities of AI models, but we would like to be clear that the IMO cannot validate the methods, including the amount of compute used or whether there was any human involvement, or whether the results can be reproduced. What we can say is that correct mathematical proofs…

Is b) really that unlikely?

Yes

Re: OpenAI claims gold-medal performance at IMO 2025

#608

Post by torginus » Sat, Jul 19, 2025, 11:03 PM UTC

tim333 wrote: post #604

torginus wrote: post #577

Earlier quoted context omitted.

Thing is, for example, all of classical physics can be derived from Newton's laws, Maxwell's equations and the laws of Thermodynamics, all of which can be written on a slip of paper. A sufficiently brilliant and determined human can invent or explain everything armed only with this knowledge. There's no need to train him on a huge corpus of text, like they do with ChatGPT. Not sure what this model's like, but I'm qui…

If you look at the history of physics I don't think it really worked like that. It took about three centuries from Newton to Maxwell because it's hard to just deduce everything from basic principles.

I think you misundertand me, I'm making some pie in the sky statement about AI being able to discover the laws of nature in an afternoon. I'm just making the observation that if you know the basic equiations, and enough math (which is about multivariate calc), you can derive every single formula in your Physics textbook (and most undergrads do as part of their education).

Since smart people can derive a lot of knowledge from a tiny set of axioms, smart AIs should be able to as well, which means you don't need to rely on a huge volume of curated information. Which means that needing to invest the internet and training on a terabyte of text might not be how these newer models are trained, and since they don't need to learn that much raw information, they might be smaller and faster.

Re: OpenAI claims gold-medal performance at IMO 2025

#609

Post by CamperBob2 » Sat, Jul 19, 2025, 11:04 PM UTC

do_not_redeem wrote: post #134

stingraycharles wrote: post #121

My issue with all these citations is that it’s all OpenAI employees that make these claims. I’ll wait to see third party verification and/or use it myself before judging. There’s a lot of incentives right now to hype things up for OpenAI.

A third party tried this experiment with publicly available models. OpenAI did half as well as Gemini, and none of the models even got bronze. https://matharena.ai/imo/

They didn't try o3-pro, which (while slow) is far in front of the competition right now.

Re: OpenAI claims gold-medal performance at IMO 2025

#610

Post by tim333 » Sat, Jul 19, 2025, 11:04 PM UTC

wds wrote: post #345

modeless wrote: post #117

Noam Brown: > this isn’t an IMO-specific model. It’s a reasoning LLM that incorporates new experimental general-purpose techniques. > it’s also more efficient [than o1 or o3] with its thinking. And there’s a lot of room to push the test-time compute and efficiency further. > As fast as recent AI progress has been, I fully expect the trend to continue. Importantly, I think we’re close to AI substantially contributing…

> I think we’re close to AI substantially contributing to scientific discovery. The new "Full Self-Driving next year"?

We have Waymo and AlphaFold.