OpenAI claims gold-medal performance at IMO 2025 • Hacker News Board

Re: OpenAI claims gold-medal performance at IMO 2025

#71

Post by causal » Sat, Jul 19, 2025, 1:45 PM UTC

exegeist wrote: post #45

z7 wrote: post #9

Some previous predictions: In 2021 Paul Christiano wrote he would update from 30% to "50% chance of hard takeoff" if we saw an IMO gold by 2025. He thought there was an 8% chance of this happening. Eliezer Yudkowsky said "at least 16%". Source: https://www.lesswrong.com/posts/sWLLdG6DWJEy3CH7n/imo-challe...

Impressive prediction, especially pre-ChatGPT. Compare to Gary Marcus 3 months ago: https://garymarcus.substack.com/p/reports-of-llms-mastering-... We may certainly hope Eliezer's other predictions don't prove so well-calibrated.

These numbers feel kind of meaningless without any work showing how he got to 16%

Re: OpenAI claims gold-medal performance at IMO 2025

#72

Post by skdixhxbsb » Sat, Jul 19, 2025, 1:45 PM UTC

Davidzheng wrote: post #53

AIPedant wrote: post #46

Earlier quoted context omitted.

If you don't have a Twitter account then x.com links are useless, use a mirror: https://xcancel.com/polynoamial/status/1946478249187377206 Anyway, that doesn't refute my point, it's just PR from a weaselly and dishonest company. I didn't say it was "IMO-specific" but the output strongly suggests specialized tooling and training, and they said this was an experimental LLM that wouldn't be released. I strongly suspect…

We can only go off their word unfortunately and they say no formal math. so I assume it's being eval'd by a verifier model instead of a formal system. There's actually some hints of this b/c geometry in Lean is not that well developed so unless they also built their own system it's hard to do it formally (though their P2 proof is by coordinate bash (computation by algebra instead of geometric construction) so it's ha…

> We can only go off their word

We’re talking about Sam Altman’s company here. The same company that started out as a non profit claiming they wanted to better the world.

Suggesting they should be given the benefit of the doubt is dishonest at this point.

Re: OpenAI claims gold-medal performance at IMO 2025

#73

Post by sigmoid10 » Sat, Jul 19, 2025, 1:45 PM UTC

z7 wrote: post #9

Some previous predictions: In 2021 Paul Christiano wrote he would update from 30% to "50% chance of hard takeoff" if we saw an IMO gold by 2025. He thought there was an 8% chance of this happening. Eliezer Yudkowsky said "at least 16%". Source: https://www.lesswrong.com/posts/sWLLdG6DWJEy3CH7n/imo-challe...

While I usually enjoy seeing these discussions, I think they are really pushing the usefulness of bayesian statistics. If one dude says the chance for an outcome is 8% and another says it's 16% and the outcome does occur, they were both pretty wrong, even though it might seem like the one who guessed a few % higher might have had a better belief system. Now if one of them had said 90% while the other said 8% or 16%, then we should pay close attention to what they are saying.

Re: OpenAI claims gold-medal performance at IMO 2025

#74

Post by johnecheck » Sat, Jul 19, 2025, 1:48 PM UTC

constantcrying wrote: post #67

johnecheck wrote: post #48

Wow. That's an impressive result, but how did they do it? Wei references scaling up test-time compute, so I have to assume they threw a boatload of money at this. I've heard talk of running models in parallel and comparing results - if OpenAI ran this 10000 times in parallel and cherry-picked the best one, this is a lot less exciting. If this is legit, then we need to know what tools were used and how the model used…

>if OpenAI ran this 10000 times in parallel and cherry-picked the best one, this is a lot less exciting. That entirely depends on who did the cherry picking. If the LLM had 10000 attempts and each time a human had to falsify it, this story means absolutely nothing. If the LLM itself did the cherry picking, then this is just akin to a human solving a hard problem. Attempting solutions and falsifying them until the des…

The key bit here is whether the LLM doing the cherry picking had knowledge of the solution. If it didn't, this is a meaningful result. That's why I'd like more info, but I fear OpenAI is going to try to keep things under wraps.

Re: OpenAI claims gold-medal performance at IMO 2025

#75

Post by gus_massa » Sat, Jul 19, 2025, 1:49 PM UTC

gniv wrote: post #44

From that thread: "The model solved P1 through P5; it did not produce a solution for P6." It's interesting that it didn't solve the problem that was by far the hardest for humans too. China, the #1 team got only 21/42 points on it. In most other teams nobody solved it.

In the IMO, the idea is that the first day you get P1, P2 and P3, and the second day you get P4, P5 and P6. Usually, ordered by difficulty, they are P1, P4, P2, P5, P3, P6. So, usually P1 is "easy" and P6 is very hard. At least that is the intended order, but sometime reality disagree.

Edit: Fixed P4 -> P3. Thanks.

Re: OpenAI claims gold-medal performance at IMO 2025

#76

Post by zkmon » Sat, Jul 19, 2025, 1:50 PM UTC

This is an awesome progress in human achievement to get these machines intelligent. And this is also a fast regress and decline on the human wisdom!

We are simply greasing the grooves and letting things slide faster and faster and calling it progress. How does this help to make the human and nature integration better?

Does this improve climate or make humans adapt better to changing climate? Are the intelligent machines a burning need for the humanity today? Or is it all about business and political dominance? At what cost? What's the fall out of all this?

Re: OpenAI claims gold-medal performance at IMO 2025

#77

Post by chairhairair » Sat, Jul 19, 2025, 1:53 PM UTC

OpenAI simply can’t be trusted on any benchmarks: https://news.ycombinator.com/item?id=42761648

Re: OpenAI claims gold-medal performance at IMO 2025

#78

Post by jebarker » Sat, Jul 19, 2025, 1:53 PM UTC

zkmon wrote: post #76

This is an awesome progress in human achievement to get these machines intelligent. And this is also a fast regress and decline on the human wisdom! We are simply greasing the grooves and letting things slide faster and faster and calling it progress. How does this help to make the human and nature integration better? Does this improve climate or make humans adapt better to changing climate? Are the intelligent machi…

Nobody knows the answers to these questions. Relying on AGI solving problems like climate change seems like a risky strategy but on the other hand it’s very plausible that these tools can help in some capacity. So we have to build, study and find out but also consider any opportunity cost of building these tools versus others.

Re: OpenAI claims gold-medal performance at IMO 2025

#79

Post by crinkly » Sat, Jul 19, 2025, 1:55 PM UTC

parsimo2010 wrote: post #68

ktallett wrote: post #19

Earlier quoted context omitted.

[flagged]

I am a professor in a math department (I teach statistics but there is a good complement of actual math PhDs) and there are only about 10% who care about these types of problems and definitely less than half who could get gold on an IMO test even if they didn’t care. They are all outstanding mathematicians, but the IMO type questions are not something that mathematicians can universally solve without preparation. The…

100% agree with this.

My second degree is in mathematics. Not only can I probably not do these but they likely aren’t useful to my work so I don’t actually care.

I’m not sure an LLM could replace the mathematical side of my work (modelling). Mostly because it’s applied and people don’t know what they are asking for, what is possible or how to do it and all the problems turn out to be quite simple really.

Re: OpenAI claims gold-medal performance at IMO 2025

#80

Post by jebarker » Sat, Jul 19, 2025, 1:56 PM UTC

ktallett wrote: post #19

dylanbyte wrote: post #16

These are high school level only in the sense of assumed background knowledge, they are extremely difficult. Professional mathematicians would not get this level of performance, unless they have a background in IMO themselves. This doesn’t mean that the model is better than them in math, just that mathematicians specialize in extending the frontier of math. The answers are not in the training data. This is not a mode…

[flagged]

IMO questions are to math as leetcode questions are to software engineering. Not necessarily easier or harder but they test ability on different axes. There’s definitely some overlap with undergrad level proof style questions but I disagree that being a working mathematician would necessarily mean you can solve these type of questions quickly. I did a PhD in pure math (and undergrad obv) and I know I’d have to spend time revising and then practicing to even begin answering most IMO questions.