Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What sort of maths are LLMs good at?

gowers.wordpress.com

Post reply on HN
Print view View original

131–140 of 189 posts

Re: What sort of maths are LLMs good at?

#131

Post by gowld »

fn-mote wrote: post #58
svantana wrote: post #29

Earlier quoted context omitted.

I would be extremely surprised if something as elegant, terse, and useful as the Fourier Transform had been missed by human mathematicians up until now. All expressible theorems are enumerable, after all (if we limit ourselves to a finite alphabet). It seems likely that any new theorems are long, highly complex and esoteric, regardless of human or machine origin.

1. The computer is going to struggle to recognize elegance. I’m not sure it’s relevant at this point (but who knows). 2. The statement about proofs is just way wrong. It doesn’t sound like you are familiar enough with them. This isn’t exactly what you implied, but witness the very short disproof of the Jacobean Conjecture.

> the very short disproof of the Jacobean Conjecture.

That's a counterexample (finding a needle in a haystack), not an elegant proof. Proving the conjecture would be elegant, if it were true but somehow still resisted proof nearly as much as the conjecture did because the conjecture was false.

Re: What sort of maths are LLMs good at?

#132

Post by CamperBob2 »

zahlman wrote: post #119
h_mirin wrote: post #5

This is really an argument about test-time scaling, even though the post never uses the term. These days "test-time scaling" mostly means letting the model talk to itself for longer, but the first genuinely surprising results came from plain sampling. Google's AlphaCode generated millions of candidate programs and filtered them down to a handful of submissions, which beat the average human programmer in 2022, before…

> Sampling is what AI is good at. You might think so, but I tried asking ChatGPT to solve one of the puzzles from https://en.wikipedia.org/wiki/Countdown_(game_show) (which a Python script can brute-force on my 12-year-old hardware in half a second) and it made an elementary arithmetic error that's decidedly not human-like.

"ChatGPT" could mean anything from Instant to Pro. What was the prompt, and which model(s) did you try?

Re: What sort of maths are LLMs good at?

#133

Post by woopsn »

scronkfinkle wrote: post #8

> A good sign that LLMs have reached human level for a much wider class of problems will be if they start proving theorems using methods that, like much of the very best human mathematics, are new and surprising but that with hindsight come to seem beautiful and natural. They should also be methods that are difficult to stumble on by accident. It is hard to say precisely what would count as such a proof, but I think…

> I could never imagine an LLM in its current form inventing something as elegant as the Fourier transform.

This is the crux imo. It doesn't really matter what absolute capability AI has at the moment, but whether we are on track with respect to architecture and training approaches. In my mind the only test for AGI is, if the thing trained up to the cut-off of Fourier's time (or Descartes, Newton etc), it should arrive at or exceed their insight.

^ As a measure of intelligence. But it's probably the case either way that LLM is more valuable in terms of the coordinated grunt work we'll put it to, than leaps of insight.

Re: What sort of maths are LLMs good at?

#134

Post by TuringTest »

wyager wrote: post #124
TuringTest wrote: post #120

Earlier quoted context omitted.

LLMs are terrible at anything systematic. They're incredibly good at anything heuristic, so it makes sense that they can explore wide mathematical spaces fast and converge towards interesting regions. But ask them to enumerate all the intermediate steps required to create a formal direct proof, and it will loose attention and forget important details as they go out of their input window size. You need to combine them…

> But ask them to enumerate all the intermediate steps required to create a formal direct proof, and it will loose attention and forget important details as they go out of their input window size. It's interesting how people will comment on LLM capabilities despite clearly not having engaged with frontier models in any meaningful way in a long time Having models write Lean proofs of mathematical claims is standard op…

Yeah but the LLM can only handle proofs that hold inside its context window. Proofs for novel theories requiring thousands of pages with dozen millions of steps will need support from external tools to organize the full structure of the formal document; it cannot be done by the LLM inference process alone, which was my point. It would be like asking a mathematician to proof theorems without pen and paper; external tooling is a must, the statistical essential nature of generating content from weights is 1) error prone and 2) not suitable for chains of systematic reasoning that are longer than the attention span.

The proofs will be only as good as the framework for linking successive instances of reasoning.

Re: What sort of maths are LLMs good at?

#135

Post by richard_chase »

calf wrote: post #113
BeetleB wrote: post #107

Earlier quoted context omitted.

> But yes, as a Fields medalist he is an expert on LLMs. Not what I was claiming, but since you are, feel free to expand on this.

I like this counterexample technique against sarcastic comments, I should try it next time someone writes a dismissive comment.

You could try it but I don't think it's going to work. Go ahead, try it.

Re: What sort of maths are LLMs good at?

#136

Post by richard_chase »

BeetleB wrote: post #107
richard_chase wrote: post #96

Earlier quoted context omitted.

That means the post was good, even though the blurry math images are illegible. But yes, as a Fields medalist he is an expert on LLMs.

> But yes, as a Fields medalist he is an expert on LLMs. Not what I was claiming, but since you are, feel free to expand on this.

The Fields medal is awarded to those who are the best at using LLMs (e.g. vibe-coding Lean) and those with the best WordPress blog with blurry math images. It is a huge honor in the vibe-coding community.

Re: What sort of maths are LLMs good at?

#137

Post by jvanderbot »

h_mirin wrote: post #5

This is really an argument about test-time scaling, even though the post never uses the term. These days "test-time scaling" mostly means letting the model talk to itself for longer, but the first genuinely surprising results came from plain sampling. Google's AlphaCode generated millions of candidate programs and filtered them down to a handful of submissions, which beat the average human programmer in 2022, before…

> we may end up with results that are correct, machine checkable, and not explainable in any way we find satisfying

sounds like quantum mechanics

Re: What sort of maths are LLMs good at?

#138

Post by zmgsabst »

steinwinde wrote: post #53

For a list of AI accomplishments in mathematics see https://mathoverflow.net/questions/502120/examples-for-the-u... - or a candidate list here: https://aimath.robertj1.com/ . Many have observed an affinity of AI to the search for counterexamples - or examples. Looking at afore lists, something much more sociological crosses my mind: There is a hunt for answering prominent, clearly stated problems. I'm not a mathemati…

You’re correct that those things are also what mathematics is about — but they’re less constrained, hence current LLMs aren’t as good at them.

However, your last question is incorrect: people are working on that, but there haven’t been hugely useful results.

But as an example, I’ve been slowly working on implementing frameworks for theory distillation — eg, take a corpus of science papers and derive a consistent model of the world from them, such as in Lean. (Or more specifically, a sheaf defining what consistent theories are possible.)

Re: What sort of maths are LLMs good at?

#139

Post by gowld »

empath75 wrote: post #70
scronkfinkle wrote: post #8

> A good sign that LLMs have reached human level for a much wider class of problems will be if they start proving theorems using methods that, like much of the very best human mathematics, are new and surprising but that with hindsight come to seem beautiful and natural. They should also be methods that are difficult to stumble on by accident. It is hard to say precisely what would count as such a proof, but I think…

IME, LLMs are primarily good at grinding through cases, which is why you see them pushing upper and lower bounds and finding counter examples. I spent a few weeks working on a number theory proof with Claude off and on and it spent hours and hours and hours grinding through one shape of polynomial after another, reporting "progress", and it's true, it proved what I was trying to prove for more and more classes of pol…

How much API usage cost (or what subscription did it fit in)?

Re: What sort of maths are LLMs good at?

#140

Post by devindotcom »

scronkfinkle wrote: post #77
root-parent wrote: post #44

Earlier quoted context omitted.

>> To me, this is more or less what I would think "Artificial General Intelligence" is So then you need to explain ARC-AGI-3: https://arxiv.org/abs/2603.24621 "Our testing shows humans can solve 100% of the environments, in contrast to frontier AI systems which, as of March 2026, score below 1%." Back 1996, EQP automatically solved the Robbins conjecture. But nobody concluded EQP was generally intelligent. https://ww…

> So then you need to explain ARC-AGI-3: https://arxiv.org/abs/2603.24621 I don't, we originally had the turing test which was designed to determine human intelligence by its ability to imitate us with natural dialogue, but we've since defeated that. I stated "to me" because it's my personal opinion on a definition whose goalpost will probably never stop being moved. > Back 1996, EQP automatically solved the Robbins…

look up the actual origin and purpose of the turing test, you may be surprised
Post reply on HN