Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What sort of maths are LLMs good at?

gowers.wordpress.com

Post reply on HN
Print view View original

61–70 of 189 posts

Re: What sort of maths are LLMs good at?

#61

Post by Anon1096 »

tcp_handshaker wrote: post #4
n4r9 wrote: post #2

A thoughtful and measured post, as usual from Gowers. The final note is neat and worth pasting out here in full: > A good sign that LLMs have reached human level for a much wider class of problems will be if they start proving theorems using methods that, like much of the very best human mathematics, are new and surprising but that with hindsight come to seem beautiful and natural. They should also be methods that ar…

>>A good sign that LLMs have reached human level for a much wider class >> of problems will be if they start proving theorems using methods that, like much of the very best human mathematics, are new and surprising but that with hindsight come to seem beautiful and natural. I must be taking crazy pills and the AGI surely will pass me by... But TODAY, middle August 2026...And in the context of testing and evaluating t…

Reading the responses to your comment the discussion would be a lot more productive if you shared your logs (preferably several of different top models since that's what you're claiming) where LLMs fail at this. Not very useful for people to go back and forth speculating on what you could have asked and with what formulation. As it stands for me simdezimon's logs are pretty definitive that there shouldn't be any problem for current capabilities agents to solve the task.

Re: What sort of maths are LLMs good at?

#62

Post by jgalt212 »

h_mirin wrote: post #5

This is really an argument about test-time scaling, even though the post never uses the term. These days "test-time scaling" mostly means letting the model talk to itself for longer, but the first genuinely surprising results came from plain sampling. Google's AlphaCode generated millions of candidate programs and filtered them down to a handful of submissions, which beat the average human programmer in 2022, before…

> Google's AlphaCode generated millions of candidate programs

The trick is avoiding the infinite monkey problem. If your problem is amenable to RL, then you probably don't even need an LLM, Monte Carlo Tree Search gets you there with less expensive hardware.

Re: What sort of maths are LLMs good at?

#64

Post by hansvm »

tcp_handshaker wrote: post #4
n4r9 wrote: post #2

A thoughtful and measured post, as usual from Gowers. The final note is neat and worth pasting out here in full: > A good sign that LLMs have reached human level for a much wider class of problems will be if they start proving theorems using methods that, like much of the very best human mathematics, are new and surprising but that with hindsight come to seem beautiful and natural. They should also be methods that ar…

>>A good sign that LLMs have reached human level for a much wider class >> of problems will be if they start proving theorems using methods that, like much of the very best human mathematics, are new and surprising but that with hindsight come to seem beautiful and natural. I must be taking crazy pills and the AGI surely will pass me by... But TODAY, middle August 2026...And in the context of testing and evaluating t…

I'm not sure why exactly, but I've heard the same from every single person using LLMs for anything related to jobs. The posting says it needs at least a B.S., and the LLM denies an applicant because they have an M.S. The posting thinks it needs 3yrs of work experience in XYZ technology, and it won't add it to the candidate's list because it doesn't have the context that the HR/LLM filter on the posting adds a bunch of nonsensical requests or that some other combination of skills makes the candidate stand out above and beyond that missing "requirement." And so on. The quality is quite poor.

On the other end of it, something like 80% of resumes I receive right now are clearly hallucinated -- referencing accomplishments that are copy-pasted from the novel-to-our-company thing in the job description a candidate will be working on, usually claiming they did XYZ at big tech a decade before the thing existed, or similarly with languages and skills. The resume "tailoring" process just manufactures lies rather than tailoring actual experience to the actual job.

Re: What sort of maths are LLMs good at?

#65

Post by user43928 »

steinwinde wrote: post #53

For a list of AI accomplishments in mathematics see https://mathoverflow.net/questions/502120/examples-for-the-u... - or a candidate list here: https://aimath.robertj1.com/ . Many have observed an affinity of AI to the search for counterexamples - or examples. Looking at afore lists, something much more sociological crosses my mind: There is a hunt for answering prominent, clearly stated problems. I'm not a mathemati…

I know nothing about mathematics, but are there not famous mathematicians like Terence Tao who utilize AI and are obviously interested in theory building?

Re: What sort of maths are LLMs good at?

#66

Post by yorwba »

YeGoblynQueenne wrote: post #40

Disclaimer: I only scanned the article quickly; I might be re-stating something already in the article. We have just got some very strong evidence about the way in which LLM-based systems solve mathematical problems and this evidence supports what many have already suspected including myself. Here's what I'm talking about. On 10 August Anthropic released an article [1] claiming that: An unreleased research version of…

> We've had the ability to brute-force all of mathematics "eventually"

In a way, yes. You can easily write a program that recursively enumerates all provable theorems in some order. But if you want a proof of a specific theorem, how do you find it in the list? You need to encode the theorem in a formal syntax first, and since mathematics is built on towers of definitions referencing other definitions, that alone is a significant amount of work before you can even write down what you want to prove.

If you want brute force alone, specialized solvers are likely a better choice than LLMs, but what LLMs add to the table is the ability to work with mathematics as it has already been written down. And even though they're bad at brute-forcing, they're still better at it than humans.

An example of a good division of labor is the SAT Attack on Tarski's High School Algebra Problem https://arxiv.org/abs/2608.08421 where they construct a formula with O(n⁴) variables and O(n⁶) clauses and use a SAT solver to show that it is unsatisfiable for n ≤ 11 but satisfiable for n = 12. Then they use an LLM to help them write a Lean proof that the SAT solver input is equivalent to the human-readable description of what they wanted to prove.

Re: What sort of maths are LLMs good at?

#68

Post by seanclayton »

scronkfinkle wrote: post #8

> A good sign that LLMs have reached human level for a much wider class of problems will be if they start proving theorems using methods that, like much of the very best human mathematics, are new and surprising but that with hindsight come to seem beautiful and natural. They should also be methods that are difficult to stumble on by accident. It is hard to say precisely what would count as such a proof, but I think…

Would you trust a bridge built by it with zero human interference? A train? A plane? A skyscraper? What about fight in a war? Unless you can, I wouldn't consider it AGI, because you're actually trusting human intelligence to verify the bridge or train or plane or skyscraper is safe or that the robot is following orders. And even then, you're trusting human-influenced guardrails etc. I would consider it AGI when an AI-created LLM can do all of these things and you trust them with your children's lives. Would you trust an AGI cop to protect your children from a violent criminal? Unless you can, believing what we have as AGI is just an empty opinion with no meaning behind it.

Re: What sort of maths are LLMs good at?

#69

Post by foco_tubi »

CodeCompost wrote: post #20
tcp_handshaker wrote: post #4

Earlier quoted context omitted.

>>A good sign that LLMs have reached human level for a much wider class >> of problems will be if they start proving theorems using methods that, like much of the very best human mathematics, are new and surprising but that with hindsight come to seem beautiful and natural. I must be taking crazy pills and the AGI surely will pass me by... But TODAY, middle August 2026...And in the context of testing and evaluating t…

LLMs have no sense of geography. They measure distances between parts of words, not distances between parts of world.

“Canada is north of the United States” has a higher probability of correctness than “Canada is east of Greenland.”

Re: What sort of maths are LLMs good at?

#70

Post by empath75 »

scronkfinkle wrote: post #8

> A good sign that LLMs have reached human level for a much wider class of problems will be if they start proving theorems using methods that, like much of the very best human mathematics, are new and surprising but that with hindsight come to seem beautiful and natural. They should also be methods that are difficult to stumble on by accident. It is hard to say precisely what would count as such a proof, but I think…

IME, LLMs are primarily good at grinding through cases, which is why you see them pushing upper and lower bounds and finding counter examples.

I spent a few weeks working on a number theory proof with Claude off and on and it spent hours and hours and hours grinding through one shape of polynomial after another, reporting "progress", and it's true, it proved what I was trying to prove for more and more classes of polynomials, but it was biting off pieces of an infinite tower of classes with no hope of closing it for _all_ polynomials.

That happens to be a good way to find counter-examples, though, and when I posed a slightly different version of my problem, it found a counter example in about 90 minutes.

And in fact, finding the counter example for the related problem allowed Claude to finally prove the thing I wanted to prove to begin with, by lifting the problem to a characteristic where that counter example didn't exist, proving my question there, and then proving that it still was equivalent to my original question.

Post reply on HN