Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What sort of maths are LLMs good at?

gowers.wordpress.com

Post reply on HN
Print view View original

21–30 of 189 posts

Re: What sort of maths are LLMs good at?

#21

Post by tcp_handshaker »

criley2 wrote: post #17
tcp_handshaker wrote: post #4

Earlier quoted context omitted.

>>A good sign that LLMs have reached human level for a much wider class >> of problems will be if they start proving theorems using methods that, like much of the very best human mathematics, are new and surprising but that with hindsight come to seem beautiful and natural. I must be taking crazy pills and the AGI surely will pass me by... But TODAY, middle August 2026...And in the context of testing and evaluating t…

[flagged]

Its the tools. Sell those RSUs while they last.

Most of these are 2026....

Frontier LLMs Still Struggle with Simple Reasoning Tasks - https://arxiv.org/abs/2507.07313

General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Tasks - https://arxiv.org/abs/2604.11778

LLMEval-Logic: A Solver-Verified Chinese Benchmark for Logical Reasoning of LLMs with Adversarial Hardening - https://arxiv.org/abs/2605.19597

LogicGraph: Benchmarking Multi-Path Logical Reasoning via Neuro-Symbolic Generation and Verification - https://arxiv.org/abs/2602.21044

Blind-Spots-Bench: Evaluating Blind Spots in Multimodal Models - https://arxiv.org/abs/2607.08317

Vision-Language Models Lag Human Performance on Physical Dynamics and Intent Reasoning - https://arxiv.org/abs/2601.01547

Do Vision-Language Models Understand 3D Scenes or Just Catalogue Objects? - https://arxiv.org/abs/2605.20448

The Reversal Curse: LLMs Trained on “A is B” Fail to Learn “B is A” - https://arxiv.org/abs/2309.12288

Large Language Model Reasoning Failures - https://arxiv.org/abs/2602.06176

Re: What sort of maths are LLMs good at?

#22

Post by pringk02 »

laszlojamf wrote: post #13
h_mirin wrote: post #5

This is really an argument about test-time scaling, even though the post never uses the term. These days "test-time scaling" mostly means letting the model talk to itself for longer, but the first genuinely surprising results came from plain sampling. Google's AlphaCode generated millions of candidate programs and filtered them down to a handful of submissions, which beat the average human programmer in 2022, before…

for somebody who's out of the loop: what's the fuss over the ABC conjecture?

https://en.wikipedia.org/wiki/Inter-universal_Teichm%C3%BCll...

Wikipedia is maybe the narrow end of a wedge into this topic but the controversy revolves around a very large and very complex paper that few people are equipped to understand and some of those who are able believe the proof is false.

Re: What sort of maths are LLMs good at?

#23

Post by xnorswap »

geon wrote: post #15
xnorswap wrote: post #11

Earlier quoted context omitted.

I found it very difficult to parse your description, "Clear, not too long not too short prompt, for LLMs to go out and research freelance consulting gigs for one specific IT domain, and in one specific country in Europe, including maybe opportunities driven from temp agencies based in geographically close countries." ( I was trying to quote a single sentence and then realised it ran on for the whole paragraph. ) Give…

That was not the prompt.

I know that. We don't know what the prompt was. We only have a self-assessment of the quality of the prompt from the person who wrote it.

It sounds like they're hitting a data source quality issue, which is hardly uncommon in scraping.

It's common for job boards to obscure who the real clients are, and if the scraping engine is LLM powered ( rather than LLM written ), then I would expect it to accidentally present agencies as the contracting organisation sometimes.

Breaking down the process so you can inspect the messy middle of a data pipeline is an important part of software engineering, but it sounds like they've tossed a messy task at an LLM and expected it to be proficient end-to-end.

Re: What sort of maths are LLMs good at?

#24

Post by steinwinde »

laszlojamf wrote: post #13
h_mirin wrote: post #5

This is really an argument about test-time scaling, even though the post never uses the term. These days "test-time scaling" mostly means letting the model talk to itself for longer, but the first genuinely surprising results came from plain sampling. Google's AlphaCode generated millions of candidate programs and filtered them down to a handful of submissions, which beat the average human programmer in 2022, before…

for somebody who's out of the loop: what's the fuss over the ABC conjecture?

This is a reference to Inter-Universal Teichmüller Theory. Its Wikipedia article gives a good overview (https://en.wikipedia.org/wiki/Inter-universal_Teichm%C3%BCll...). In maths lasting disagreements over a published "proof" are rare, but IUTT is an example of it. What the article misses: There is a more recent, ongoing effort to formalize the published proof in Lean under the name of "LANA" (e.g. see https://zen.ac.jp/news/zmcpostevent0717e and https://github.com/katobungen/LANA_report_202607/blob/pdf/LA... for a recent update). I guess most mathematicians agree that a successful compile of the proof in Lean would confirm its validity. My personal impression is that the process got stuck at the very point Peter Scholze and Jakob Stix pointed out 8 years ago. Officially LANA has still not reached a conclusion.

Re: What sort of maths are LLMs good at?

#25

Post by porridgeraisin »

scronkfinkle wrote: post #8

> A good sign that LLMs have reached human level for a much wider class of problems will be if they start proving theorems using methods that, like much of the very best human mathematics, are new and surprising but that with hindsight come to seem beautiful and natural. They should also be methods that are difficult to stumble on by accident. It is hard to say precisely what would count as such a proof, but I think…

I share the same thinking. What do you think is a good way to try to define this "elegance"? If we try to use the mental framework of

Step 1. LLM "brute forces" a search

Step 2. We train on this trace

Step 3. In the next model, LLM internally makes a "shortcut" for this path and "brute forces" it quicker (or one shots its in the best case)

And we want to ultimately show why that definitio evades this framework.

Re: What sort of maths are LLMs good at?

#26

Post by criley2 »

tcp_handshaker wrote: post #21
criley2 wrote: post #17

Earlier quoted context omitted.

[flagged]

Its the tools. Sell those RSUs while they last. Most of these are 2026.... Frontier LLMs Still Struggle with Simple Reasoning Tasks - https://arxiv.org/abs/2507.07313 General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Tasks - https://arxiv.org/abs/2604.11778 LLMEval-Logic: A Solver-Verified Chinese Benchmark for Logical Reasoning of LLMs with Adversarial Hardening - htt…

What a sloppy reply. You've hijacked a thread on mathematics first to complain that your incompetent attempt to use ChatGPT to find a job failed, but it seems now that this was a ruse to instead begin arguments unrelated to the article at all where you just spam arxiv links you've never read to "prove" that AI is a scam.

This comes across, frankly, as either Dunning-Kruger (classic illusory superiority), or potentially as mental illness. The slop dump is highly reminiscent of how a schizophrenic friend of mine communicates.

Do you really think slopping down a bunch of random arxiv links "proves" that AI is a scam and you're so smart and everyone else isn't?

Most awkwardly for your arxiv slop -- most of this is irrelevant to your central claim, and you've missed papers that are much closer.

For example your LogicGraph paper: "Can't exhaustively enumerate all minimal proofs" is not "can't distinguish Ireland from London".

Or your "Do VLMs Understand 3D Scenes..." is nothing more than citation decoration, completely irrelevant to our discussion.

Or your "Frontier LLMs Still Struggle with Simple Reasoning Tasks" which is potentially your pièce de résistance, it supports brittle multi-step constraint handling, but isn't remotely an eval of a modern web-search agent.

For example, VibeSearchBench would have been far more relevant to your claims https://arxiv.org/html/2605.27882v1 (but still obviously not proof that AI is "a parlour trick")

Going further: my point that we need to discuss your beginner's approach to the harness is substantiated clearly here: https://arxiv.org/html/2605.23950v1

Finally, failure to exhibit human-like generality is not evidence of absence of intelligence. It is evidence that whatever cognitive machinery LLMs possess has a very different error distribution from ours. Your General365, LLMEval-Logic and the Reversal Curse are actually fascinating evidence for that jaggedness, rather than proof of your claim that AI is a scam.

Re: What sort of maths are LLMs good at?

#27

Post by wizzwizz4 »

criley2 wrote: post #17
tcp_handshaker wrote: post #4

Earlier quoted context omitted.

>>A good sign that LLMs have reached human level for a much wider class >> of problems will be if they start proving theorems using methods that, like much of the very best human mathematics, are new and surprising but that with hindsight come to seem beautiful and natural. I must be taking crazy pills and the AGI surely will pass me by... But TODAY, middle August 2026...And in the context of testing and evaluating t…

[flagged]

> Try some context engineering. Try customizing your harness. Try having the harness improve itself. These are AI 101 lessons

If AI is really that complicated, it sounds like it would be easier to write an ordinary computer program to aggregate job boards.

Re: What sort of maths are LLMs good at?

#28

Post by xnorswap »

tcp_handshaker wrote: post #19
xnorswap wrote: post #11

Earlier quoted context omitted.

I found it very difficult to parse your description, "Clear, not too long not too short prompt, for LLMs to go out and research freelance consulting gigs for one specific IT domain, and in one specific country in Europe, including maybe opportunities driven from temp agencies based in geographically close countries." ( I was trying to quote a single sentence and then realised it ran on for the whole paragraph. ) Give…

You can easily test this yourself with the SOTA models....or read the corroborating literature... "General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Tasks" https://arxiv.org/abs/2604.11778 "...General365, a benchmark specifically designed to assess general reasoning in LLMs. By restricting background knowledge to a K-12 level, General365 explicitly decouples reasoning…

What proportion of the human population could answer the example from that paper?

    Question: Strangers A, B, C, D, and E line up from youngest on the left to oldest on the right. Their clothing
    colors and shoe colors all differ, and they come from five different regions.
    Known facts:
    1. A is from Morocco.
    2. D is five years older than B.
    3. E is older than A.
    4. C stands next to D.
    5. A stands next to B.
    6. The person in teal shoes is not adjacent to the person from Vanuatu.
    7. One twelve-year-old wears yellow shoes.
    8. The person in orange shoes wears white clothing.
    9. The person in blue clothing is from Chile.
    10. The youngest person wears red shoes.
    11. Counting from the right, the fourth person comes from South Africa.
    12. E wears yellow clothing.
    13. The person in green shoes does not wear multicolored clothing.
    14. Two people are twelve years old, ordered by birth month.
    15. One adult is thirty-five years old, and that age is sixteen less than the combined ages of the other four.
    If you multiply every possible age C might have, what number do you obtain?
What does "The person in green shoes does not wear multicolored clothing" even mean?

Nowhere is "multicoloured" defined, are we to assume it should be treated as a colour and implied that someone else must be wearing "multicoloured clothing"? Because strictly that doesn't logically follow, and it ought to be phrased as "The person in green shoes is not the person wearing multicoloured clothing" if that is the case.

This is an extremely hard logic puzzle, especially since it's revealed at the end that there are multiple solutions.

I'd expect anyone to struggle unless armed with prolog.

Re: What sort of maths are LLMs good at?

#29

Post by svantana »

scronkfinkle wrote: post #8

> A good sign that LLMs have reached human level for a much wider class of problems will be if they start proving theorems using methods that, like much of the very best human mathematics, are new and surprising but that with hindsight come to seem beautiful and natural. They should also be methods that are difficult to stumble on by accident. It is hard to say precisely what would count as such a proof, but I think…

I would be extremely surprised if something as elegant, terse, and useful as the Fourier Transform had been missed by human mathematicians up until now. All expressible theorems are enumerable, after all (if we limit ourselves to a finite alphabet). It seems likely that any new theorems are long, highly complex and esoteric, regardless of human or machine origin.

Re: What sort of maths are LLMs good at?

#30

Post by criley2 »

wizzwizz4 wrote: post #27
criley2 wrote: post #17

Earlier quoted context omitted.

[flagged]

> Try some context engineering. Try customizing your harness. Try having the harness improve itself. These are AI 101 lessons If AI is really that complicated, it sounds like it would be easier to write an ordinary computer program to aggregate job boards.

I totally agree - designing a competent AI agent with a fully customized harness to successfully pull off this task is a much more challenging engineering effort than merely creating an ordinary computer program. Had OP made chatgpt write an ordinary program instead, they likely would have succeeded in their task.
Post reply on HN