Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Epoch confirms GPT5.4 Pro solved a frontier math open problem

epoch.ai

Post reply on HN
Print view View original

711–720 of 744 posts

Re: Epoch confirms GPT5.4 Pro solved a frontier math open problem

#711

Post by famouswaffles »

staticassertion wrote: post #543
famouswaffles wrote: post #502

Earlier quoted context omitted.

>But that does not mean that I have to think that the appearance of intelligence always is intelligence, or that an LLM/ Agent is doing what humans do. You can think whatever you want, but an untestable distinction is an imaginary one.

First of all, that's not true. Not every position has to be empirically justified. I can reason about a position in all sorts of ways without testing. Here's an obvious example that requires no test at all: 1. Functional properties seem to arise from structural properties 2. Brains and LLMs have radically different structural properties 3. Two constructs with radically, fundamentally different structural properties a…

Your reasoning may lower your confidence, but until it connects to observable differences, it is still at least partly a story you are telling yourself.

More importantly, the question is not whether LLMs work the same way human brains do. You may care about that, but many people do not. The relevant question is whether they exhibit the functional properties we care about. Saying “they are structurally different, therefore not really intelligent” is a lot like insisting planes are not really flying because they do not flap like birds.

And on your last point: in practice, it is not testable. There is no decisive intelligence test that sorts all humans into one bucket and all LLMs into another. So if your distinction cannot be cashed out behaviorally, functionally, or empirically, then it starts to look less like a serious difference and more like a metaphysical preference.

Re: Epoch confirms GPT5.4 Pro solved a frontier math open problem

#712

Post by aspenmartin »

Gareth321 wrote: post #689
aspenmartin wrote: post #558

Earlier quoted context omitted.

this is true because markets are generally efficient. It's very hard to find predictive signals. That is a completely different space than what we're talking about here. Performance is incredibly predictable through scaling laws that continue to hold even at the largest scales we've built

I agree this is a new space and prediction volatility is much higher. We have evidence going back to at least 2019 that improvements have been exponential ( https://metr.org/blog/2025-03-19-measuring-ai-ability-to-com... ). The benchmarks are all over the place because improvements don't happen in a straight line. Even composites aren't that useful because the last 10% improvement can require more effort than the fir…

I agree with all of this -- the one nit I'll say is that scaling laws (e.g. Chinchilla -- classic paper on this that still holds) are based on next-token log loss on an evaluation set for pretraining, and follow (empirically) very consistent powerlaw relationships with compute / data (there is an ideal mixture of compute + data, and the thing you scale is the compute at this ideal mixture). So that's all I mean by performance -- we do also have as you observe benchmark performance trends (which are measured on the final model, after post-training, RL stages etc). These follow less predictable relationships, but it's the pretraining loss that dominates anyway.

I agree with all of this though

Re: Epoch confirms GPT5.4 Pro solved a frontier math open problem

#713

Post by azakai »

zamalek wrote: post #684
esafak wrote: post #665

Earlier quoted context omitted.

Mathematics is not the only kind of reasoning, so your conclusion is false. The human brain also has compartments for different types of activities. Why shouldn't an AI be able to use tools to augment its intelligence?

I used the mathematics example only because the GP did. There are many other examples of non-reasoning, including some papers (as recent as Feb).

There are many examples of current limitations, but do you see a reason to think they are fundamental limitations? (I'm not saying they aren't, I'm curious what the evidence is for that.)

Re: Epoch confirms GPT5.4 Pro solved a frontier math open problem

#714

Post by lern_too_spel »

zamalek wrote: post #685
throwawayk7h wrote: post #671

Earlier quoted context omitted.

I thought it might do better if I asked it to do long-form multiplication specifically rather than trying to vomit out an answer without any intermediate tokens. But surprisingly, I found it doesn't do much better.

Other comments indicate that asking it to do long multiplication does work, but the varying results makes sense: LLMs are probabilistic, you probably rolled an unlikely result.

Specifically, you need to use a reasoning model. Applying more test time compute is analogous to Kahneman's System 2 thinking, while directly taking the first output of an LLM is analogous to System 1.

This is true for solving difficult novel problems as well, with the addition of tools that an agent can use to research the problem autonomously.

Re: Epoch confirms GPT5.4 Pro solved a frontier math open problem

#715

Post by CamperBob2 »

jibal wrote: post #682
CamperBob2 wrote: post #666

Earlier quoted context omitted.

Furthermore, the LLM isn't doing things "in its head" - the headline feature of GPT LLMs is attention across all previous tokens, all of its "thoughts" are on paper LOL, talk about special pleading. Whatever it takes to reshape the argument into one you can win, I guess... LLMs don't reason. Let's see you do that multiplication in your head. Then, when you fail, we'll conclude you don't reason. Sound fair?

The conclusion that LLMs don't reason is not a consequence of them not being able to do arithmetic, so your argument isn't valid. Also, see https://news.ycombinator.com/newsguidelines.html "Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes. Comments should get more thoughtful and substantive, not less, as a topic gets more divisive. When disagreeing, please reply to the argument inste…

Comments should get more thoughtful and substantive

Yes, they should, but instead we're stuck with the stochastic-parrot crowd, who log onto HN and try their best to emulate a stochastic parrot.

Re: Epoch confirms GPT5.4 Pro solved a frontier math open problem

#716

Post by CamperBob2 »

jibal wrote: post #695
IshKebab wrote: post #688

Earlier quoted context omitted.

Plenty of humans can't do arithmetic. Can they also not reason. Reasoning isn't a binary switch. It's a multidimensional continuum. AI can clearly reason to some extent even if it also clearly doesn't reason in the same way that a human would.

> Plenty of humans can't do arithmetic. Can they also not reason. I just pointed out that this isn't valid reasoning ... it's a fallacy of denial of the antecedent. No one is arguing that because LLMs can't do arithmetic, therefore they can't reason. After all, zamalek said that he can't quickly multiply large numbers in his head, but he isn't saying that therefore he can't reason. > Reasoning isn't a binary switch.…

Indeed, and a lot of humans are very bad at it, as is clear from the comments I'm responding to.

This is your idea of "conversing curiously" and "editing out swipes," I suppose.

I've had this argument too many times ... it never goes anywhere. So I won't respond again ... over and out.

A real reasoning entity might pause for self-examination here. Maybe run its chain of thought for a few more iterations, or spend some tokens calling research tools. Just to probe the apparent mismatch between its own priors and those of "a lot of humans," most of whom are not, in fact, morons.

Re: Epoch confirms GPT5.4 Pro solved a frontier math open problem

#717

Post by digitalPhonix »

dirasieb wrote: post #696
digitalPhonix wrote: post #681

Earlier quoted context omitted.

I can do it with a scratch pad. And I can also tell you when the calculation exceeds what I can do in my head and when I need a scratch pad. I can also check a long multiplication answer in my head (casting 9s, last digit etc.) and tell if there’s a mistake. The LLMs also have access to a scratch pad. And importantly don’t know when they need to use it (as in, they will sometimes get long multiplication right if you…

> And importantly don’t know when they need to use it patently false, but hey at least you’re able to see the parallel between you with a scratch pad and an LLM with a python terminal

Sure, lets test that:

https://chatgpt.com/s/t_69c420f3118081919cf525123e39598c

https://chatgpt.com/s/t_69c4215daeb481919fdaf22498fb0c4f

Do you have a different definition of false? I'm referring to their reasoning context as their scratch pad if that wasn't clear.

Re: Epoch confirms GPT5.4 Pro solved a frontier math open problem

#718

Post by zamalek »

azakai wrote: post #713
zamalek wrote: post #684

Earlier quoted context omitted.

I used the mathematics example only because the GP did. There are many other examples of non-reasoning, including some papers (as recent as Feb).

There are many examples of current limitations, but do you see a reason to think they are fundamental limitations? (I'm not saying they aren't, I'm curious what the evidence is for that.)

It's because of how transformers work, especially the fact that the output layer is a bunch of weights which we quite literally do a weighted random choice from. My hunch is that diffusion models would have a higher chance of doing real reasoning - or something like a latent space for reasoning.

Thinking that LLMs are intelligent arises from an incomplete understanding of how they work or, alternatively, having shareholders to keep happy.

Re: Epoch confirms GPT5.4 Pro solved a frontier math open problem

#719

Post by zamalek »

dirasieb wrote: post #696
digitalPhonix wrote: post #681

Earlier quoted context omitted.

I can do it with a scratch pad. And I can also tell you when the calculation exceeds what I can do in my head and when I need a scratch pad. I can also check a long multiplication answer in my head (casting 9s, last digit etc.) and tell if there’s a mistake. The LLMs also have access to a scratch pad. And importantly don’t know when they need to use it (as in, they will sometimes get long multiplication right if you…

> And importantly don’t know when they need to use it patently false, but hey at least you’re able to see the parallel between you with a scratch pad and an LLM with a python terminal

The context is the scratch pad. LLMs have perfect recall (ignoring "lost in the middle") across the entire context, unlike humans. LLMs "think on paper."

Re: Epoch confirms GPT5.4 Pro solved a frontier math open problem

#720

Post by thaumasiotes »

scoofy wrote: post #694
thaumasiotes wrote: post #692

Earlier quoted context omitted.

A different set of "everything" will fall out of the new axioms. You can enumerate it, but no one will care. That doesn't make for a perfect field, or even a good field, for machine learning to thrive in; what we care about is finding useful results. Starting with arbitrary axioms is a good way to prevent that from happening. Compare this discussion from an algebra textbook I've been reading recently: ----- The possi…

I'm talking about a deductive framework based on by definition arbitrary axioms. You're talking about an inductive framework in which we create deductive frameworks that model the world we live in as best as we can tell. This is very, very hard work. The flipping back and forth between the inductive framework and the deductive framework -- between modeling reality and discovering and testing new aspects -- is the hea…

> I'm just talking about that deductive framework, that exists, locked in it's by definition arbitrary axioms. It's in that framework that machine learning should thrive, because all of the propositions possible will fall out of the axioms.

You like to repeat yourself. Are there other words you can use to describe what you're supposedly thinking? What does it mean for machine learning to thrive in a space?

In the same way that you can derive all valid proofs from a set of axioms, you can derive all valid 320x240 bitmaps from a (much simpler!) set of axioms. Does this mean that artwork is "a perfect field for machine learning to thrive [in]"? What would be an example of a field that isn't "perfect for machine learning to thrive in"?

Post reply on HN