OpenAI claims gold-medal performance at IMO 2025

Post by yobbo » Sat, Jul 19, 2025, 2:53 PM UTC

Invictus0 wrote: post #58

npinsker wrote: post #31

Earlier quoted context omitted.

The trouble is, getting an IMO gold medal is much easier (by frequency) than being the #1 Go player in the world, which was achieved by AI 10 years ago. I'm not sure it's enough to just gesture at the task; drilling down into precisely how it was achieved feels important. (Not to take away from the result, which I'm really impressed by!)

The "AI" that won Go was Monte Carlo tree search on a neural net "memory" of the outcome of millions of previous games; this is a LLM solving open ended problems. The tasks are hardly even comparable.

Post by kelipso » Sat, Jul 19, 2025, 2:54 PM UTC

YeGoblynQueenne wrote: post #124

modeless wrote: post #117

Noam Brown: > this isn’t an IMO-specific model. It’s a reasoning LLM that incorporates new experimental general-purpose techniques. > it’s also more efficient [than o1 or o3] with its thinking. And there’s a lot of room to push the test-time compute and efficiency further. > As fast as recent AI progress has been, I fully expect the trend to continue. Importantly, I think we’re close to AI substantially contributing…

How is a claim , "clear evidence" to anything?

Post by rvz » Sat, Jul 19, 2025, 2:58 PM UTC

softwaredoug wrote: post #136

mikert89 wrote: post #133

The cynicism/denial on HN about AI is exhausting. Half the comments are some weird form of explaining away the ever increasing performance of these models I've been reading this website for probably 15 years, its never been this bad. many threads are completely unreadable, all the actual educated takes are on X, its almost like there was a talent drain

Probably because both sides have strong vested interests and it’s next to impossible to find a dispassionate point of view. The Pro AI crowd, VC, tech CEOs etc have strong incentive to claim humans are obsolete. Many tech employees see threats to their jobs and want to poopoo any way AI could be useful or competitive.

Post by grillitoazul » Sat, Jul 19, 2025, 3:01 PM UTC

grillitoazul wrote: post #112

sigmoid10 wrote: post #73

Earlier quoted context omitted.

While I usually enjoy seeing these discussions, I think they are really pushing the usefulness of bayesian statistics. If one dude says the chance for an outcome is 8% and another says it's 16% and the outcome does occur, they were both pretty wrong, even though it might seem like the one who guessed a few % higher might have had a better belief system. Now if one of them had said 90% while the other said 8% or 16%,…

From a mathematical point of view there are two factors: (1) Initial prior capability of prediction from the human agents and (2) Acceleration in the predicted event. Now we examine the result under such a model and conclude that: The more prior predictive power of human agents imply the more a posterior acceleration of progress in LLMs (math capability). Here we are supposing that the increase in training data is no…

Post by bluecalm » Sat, Jul 19, 2025, 3:01 PM UTC

stingraycharles wrote: post #119

aubanel wrote: post #109

Earlier quoted context omitted.

- AI competing is "wholly unfair" - "[AI is] far away from being substantially being better than MCTs" ^ pick only one

Yeah it’s a completely fair playing field, it’s completely obvious that AI should be able to compete with humans in the same way that robotics and computers can compete with humanity (and are better suited for many tasks). Whether or not they’re far away from being better than humans is up to debate, but the entire point of these types of benchmarks it to compare them to humans.