Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI claims gold-medal performance at IMO 2025

twitter.com

Post reply on HN
Print view View original

511–520 of 737 posts

Re: OpenAI claims gold-medal performance at IMO 2025

#511

Post by hyghjiyhu »

conartist6 wrote: post #476
lordnacho wrote: post #202

Earlier quoted context omitted.

> These models cannot reason Not trying to be a smarty pants here, but what do we mean by "reason"? Just to make the point, I'm using Claude to help me code right now. In between prompts, I read HN. It does things for me such as coding up new features, looking at the compile and runtime responses, and then correcting the code. All while I sit here and write with you on HN. It gives me feedback like "lock free message…

I think the biggest hint that the models aren't reasoning is that they can't explain their reasoning. Researchers have shown for explained that how a model solves a simple math problem and how it claims to have solved it after the fact have no real correlation. In other words there was only the appearance of reasoning.

People can't explain their reasoning either. People do a parallel construction of logical arguments for a conclusion they already reached intuitively in a way they have no clue how it happened. "The idea just popped into my head while showering" to our credit, if this post-hoc rationalization fails we are able to change our opinion to some degree.

Re: OpenAI claims gold-medal performance at IMO 2025

#512

Post by mlyle »

zer00eyz wrote: post #475
mlyle wrote: post #335

Earlier quoted context omitted.

I don't think this is a reasonable take. Some people/organizations send signals about things that we're not ready to fully drop it on the world. Others consider those signals in context (reputation of sender, prior probability of being true, reasons for sender to be honest vs. deceptive, etc). When my wife tells me there's a pie in the oven and it's smelling particularly good, I don't demand evidence or disbelieve th…

> Some people/organizations send signals about things that we're not ready to fully drop it on the world. This is called marketing. > When my wife tells me there's a pie in the oven and it's smelling particularly good, I don't demand evidence Because you have evidence, it smells. And if later your ask your wife "where is the pie" and she says "I sprayed pie scent in the air, I was just singling" how are you going to…

> This is called marketing.

Many signals are marketing, but the purpose of signals is not purely to develop markets. We all have to determine what we think will happen next and how others will act.

> Because you have evidence, it smells.

I think you read that differently than what I intended to write -- she claims it smells good.

> Open AI spent its "fool us once" card already.

> > This is from OpenAI. Here they've not been so great with public communications in the past, and they have a big incentive in a crowded marketplace to exaggerate claims.

Re: OpenAI claims gold-medal performance at IMO 2025

#513

Post by gettingoverit »

tedsanders wrote: post #361
gettingoverit wrote: post #312

Earlier quoted context omitted.

Implying results are fraudulent is completely fair when it is a fraud. The previous time they had claims about solving all of the math right there and right then, they were caught owning the company that makes that independent test , and could neither admit nor deny training on closed test set.

Just to quickly clarify: - OpenAI doesn't own Epoch AI (though they did commission Epoch to make the eval) - OpenAI denied training on the test set (and further denied training on FrontierMath-derived data, training on data targeting FrontierMath specifically, or using the eval to pick a model checkpoint; in fact, they only downloaded the FrontierMath data after their o3 training set was frozen and they didn't look a…

One piece of evidence I have is that most powerful reasoning models can answer at most 10% of my questions on PhD-student level computer science, and are unable to produce correct implementations for basic algorithms, being provided direct references to their implementations and materials that describe them. Damn, o3 can't draw an SVG arrow. Recent advancement in counting "r" in "strawberry" is basically as far as it goes.

I don't know what exactly is at play here, and how exactly OpenAI's models can produce those "exceptionally good" results in benchmarks and at the same time be utterly unable to do even a quarter of that in private evaluation of pretty much everyone I knew. I'd expect them to use some kind of RAG techniques that make the question "what was in the training set at model checkpoint" irrelevant.

If you consider that several billion dollars of investment and national security are at stake, "weird conspiracy" becomes a regular Tuesday.

Unfortunately I can't see beyond the first message of that primary source.

Re: OpenAI claims gold-medal performance at IMO 2025

#514

Post by gitfan86 »

colinplamondon wrote: post #504
ysavir wrote: post #500

Earlier quoted context omitted.

Less that AI is cheating and more that we basically found a way to take the thousand monkeys with infinite time scenario and condense that into a reasonable(?) amount of time and with some decent starting instructions. The AI wouldn't have done any of the heavy lifting of the discovery, it just iterated on the work of past researchers at speeds beyond human.

Honest question - how is that not true of those past researchers? IE, they... - Start with the context window of prior researchers. - Set a goal or research direction. - Engage in chain of thought with occasional reality-testing. - Generate an output artifact, reviewable by those with appropriate expertise, to allow consensus reality to accept or reject their work.

AI sceptics are on a trajectory towards philosophy.

Re: OpenAI claims gold-medal performance at IMO 2025

#515

Post by ben_w »

sailingparrot wrote: post #256
z7 wrote: post #9

Some previous predictions: In 2021 Paul Christiano wrote he would update from 30% to "50% chance of hard takeoff" if we saw an IMO gold by 2025. He thought there was an 8% chance of this happening. Eliezer Yudkowsky said "at least 16%". Source: https://www.lesswrong.com/posts/sWLLdG6DWJEy3CH7n/imo-challe...

Off topic, but am I the only one getting triggered every time I see a rationalist quantify their prediction of the future with single digit accuracy? It's like their magic way of trying to get everyone to forget that they reached their conclusion in completely hand-wavy way, just like every other human being. But instead of saying "low confidence" or "high confidence" like the rest of us normies, they will tell you t…

> But instead of saying "low confidence" or "high confidence" like the rest of us normies

To add to what tedsanders wrote: there's also research that shows verbal descriptions, like those, mean wildly different things from one person to the next: https://lettersremain.com/perceptions-of-probability-and-num...

Re: OpenAI claims gold-medal performance at IMO 2025

#516

Post by bamboozled »

lordnacho wrote: post #202
parasubvert wrote: post #108

Earlier quoted context omitted.

I think the main hesitancy is due to rampant anthropomorphism. These models cannot reason, they pattern match language tokens and generate emergent behaviour as a result. Certainly the emergent behaviour is exciting but we tend to jump to conclusions as to what it implies. This means we are far more trusting with software that lacks formal guarantees than we should be. We are used to software being sound by default b…

> These models cannot reason Not trying to be a smarty pants here, but what do we mean by "reason"? Just to make the point, I'm using Claude to help me code right now. In between prompts, I read HN. It does things for me such as coding up new features, looking at the compile and runtime responses, and then correcting the code. All while I sit here and write with you on HN. It gives me feedback like "lock free message…

I'm having actual coding conversations that I used to only have with senior devs, right now, while browsing HN, and code that does what I asked is being produced.

I’m using Opus 4 for coding and there is no way that model demonstrates any reasoning or demonstrates any “intelligence” in my opinion. I’ve been through the having conversations phase etc but doesn’t get you very far, better to read a book.

I use these models to help me type less now, that’s it. My prompts basically tell it to not do anything fancy and that works well.

Re: OpenAI claims gold-medal performance at IMO 2025

#517

Post by lordnacho »

conartist6 wrote: post #476
lordnacho wrote: post #202

Earlier quoted context omitted.

> These models cannot reason Not trying to be a smarty pants here, but what do we mean by "reason"? Just to make the point, I'm using Claude to help me code right now. In between prompts, I read HN. It does things for me such as coding up new features, looking at the compile and runtime responses, and then correcting the code. All while I sit here and write with you on HN. It gives me feedback like "lock free message…

I think the biggest hint that the models aren't reasoning is that they can't explain their reasoning. Researchers have shown for explained that how a model solves a simple math problem and how it claims to have solved it after the fact have no real correlation. In other words there was only the appearance of reasoning.

Is this true though? I've suggested things that it pushed back on. Feels very much like a dev. It doesn't just dumbly do what I tell it.

Re: OpenAI claims gold-medal performance at IMO 2025

#518

Post by jononor »

CamperBob2 wrote: post #437
jononor wrote: post #399

Earlier quoted context omitted.

A) We are out of the Internet-scale-for-free data. Of course the companies deploying LLM based systems at massive scale are of course ingesting a lot of human data from their users, that they are seeking to use to further improve their models. B) Has learning though "self-play" (like with AlphaZero etc) been demonstrated working for improving LLMs? What is the latest key research on this?

Certainly the models have orders of magnitude more data available to them than the smartest human being who ever lived does/did. So we can assume that if the goal is "merely" superhuman intelligence, data is not a problem. It might be a constraint on the evolution of godlike intelligence, or AGI. But at that point we're so far out in bong-hit territory that it will be impossible to say who's right or wrong about what…

Superhuman capability within the tasks that are well represented in the dataset, yes. If one takes the view that intelligence is the ability to solve novel problems (ref F. Chollet), then the amount of data alone might not take us to superhuman intelligence. At least without new breakthroughs in the construction of models or systems.

R1 managed to replicate a model on the level of one one they had access to. But as far as I know they did not improve on its predictive performance? They did improve in inference time, but that is another thing. The ability to replicate a model is well demonstrated and quite common practice for some years already, see teacher-student distillation.

Re: OpenAI claims gold-medal performance at IMO 2025

#519

Post by bamboozled »

logicchains wrote: post #158
parasubvert wrote: post #108

Earlier quoted context omitted.

I think the main hesitancy is due to rampant anthropomorphism. These models cannot reason, they pattern match language tokens and generate emergent behaviour as a result. Certainly the emergent behaviour is exciting but we tend to jump to conclusions as to what it implies. This means we are far more trusting with software that lacks formal guarantees than we should be. We are used to software being sound by default b…

>I think the main hesitancy is due to rampant anthropomorphism. These models cannot reason, they pattern match language tokens and generate emergent behaviour as a result This is rampant human chauvinism. There's absolutely no empirical basis for the statement that these models "cannot reason", it's just pseudoscientific woo thrown around by people who want to feel that humans are somehow special. By pretty much ever…

I’ve used these AI tools for multiple hours a day for months. Not seeing the reasoning party honestly. I see the heuristics part.

Re: OpenAI claims gold-medal performance at IMO 2025

#520

Post by Dilettante_ »

fnordpiglet wrote: post #95
johnecheck wrote: post #48

Wow. That's an impressive result, but how did they do it? Wei references scaling up test-time compute, so I have to assume they threw a boatload of money at this. I've heard talk of running models in parallel and comparing results - if OpenAI ran this 10000 times in parallel and cherry-picked the best one, this is a lot less exciting. If this is legit, then we need to know what tools were used and how the model used…

Why is that less exciting? A machine competing in an unconstrained natural language difficult math contest and coming out on top by any means is breath taking science fiction a few years ago - now it’s not exciting? Regardless of the tools for verification or even solvers - why is the goal post moving so fast? There is no bonus for “purity of essence” and using only neural networks. We live in an era where it’s hard…

>Why is that less exciting?

Because if I have to throw 10000 rocks to get one in the bucket, I am not as good/useful of a rock-into-bucket-thrower as someone who gets it in one shot.

People would probably not be as excited about the prospect of employing me to throw rocks for them.

Post reply on HN