[deleted]
What sort of maths are LLMs good at?
161–170 of 189 posts
Re: What sort of maths are LLMs good at?
#162Earlier quoted context omitted.
What proportion of the human population could answer the example from that paper? Question: Strangers A, B, C, D, and E line up from youngest on the left to oldest on the right. Their clothing colors and shoe colors all differ, and they come from five different regions. Known facts: 1. A is from Morocco. 2. D is five years older than B. 3. E is older than A. 4. C stands next to D. 5. A stands next to B. 6. The person…
No offense to you, but we had to do these kind of exercises in grade school on days when a sub was in. You'd get together in a group and work through the logic puzzle on a structured form (i.e., it let you represent what you know about the person wearing green shoes, or the person from Morocco, etc.)
I'm familiar with the format, I grew up doing these, published monthly under the name "Logic Problems" in the UK.
My issue with this particular puzzle was the ambiguity of that particular clue, it's as if it got re-written or badly translated, because clue 13 as written carries no information.
Re: What sort of maths are LLMs good at?
#163A thoughtful and measured post, as usual from Gowers. The final note is neat and worth pasting out here in full: > A good sign that LLMs have reached human level for a much wider class of problems will be if they start proving theorems using methods that, like much of the very best human mathematics, are new and surprising but that with hindsight come to seem beautiful and natural. They should also be methods that ar…
> with hindsight come to seem beautiful and natural. …because they’ve been internalized. I wonder if the author has ever questioned where his notions of “beauty” and “natural” come from.
I'm sure that he has, and I think that his point is solid. Mathematical beauty often has something to do with the transition from "surprising" to "natural".
Re: What sort of maths are LLMs good at?
#164Earlier quoted context omitted.
This is not "brute-force" though. It's an iterative search algorithm. You learn things at each iteration. You also don't search blindly. You use "something" (heuristics, experience, intuition) to come up with "ideas" at each iteration. You don't try 650 random programs. You try 650 different ideas each learning from the results of previous trials. Yes this is the "Universal Problem Solving Algorithm". It's actually t…
>> This is not "brute-force" though. It's an iterative search algorithm. You learn things at each iteration. You also don't search blindly. You use "something" (heuristics, experience, intuition) to come up with "ideas" at each iteration. You don't try 650 random programs. You try 650 different ideas each learning from the results of previous trials. But, learn what? All those ideas where wrong. How does an LLM "lear…
>> In this case, by happy accident, what stuck was an improved bound of the zeroes of the zeta function etc.
I’m not familiar with this specific example (or Mathematics) but I assume they keep intermediate results (python functions, lemmas, computations, intermediate proofs etc.) and formulate and explore adjacent ideas. Even with a failed attempt you can learn things. LLMs make a difference here because they can evaluate an experiment and hypothesize what went wrong or what should stick. So the search space is dynamically evolving unlike pre-LLM algorithms.
Karpathy’s Autoresearch provides a proof of concept for this method.
Re: What sort of maths are LLMs good at?
#165Earlier quoted context omitted.
Not in any useful way, though. In the Curry-Howard correspondence, being able to write a function of type "int -> string" proves that there is at least one valid string or there are no valid ints. The code of the function is a proof of this statement. For example you might prove directly that there is a string, by writing return "hello"; or you might write return bool2string(int2bool(myInt)); thus chaining two implic…
> Not in any useful way, though. Theorem provers and proof assistants like Lean, Coq/Rocq, Agda, Idris, NuPRL and Epigram all fundamentally depend on Curry-Howard. And again, the point is not that there’s some magical isomorphism, but that if LLMs are good at coding, they’re also likely to be good at certain classes of proof.
No they don't. They act on proofs, not on stupid conversions of proofs into useless programs.
Re: What sort of maths are LLMs good at?
#166Earlier quoted context omitted.
You can easily test this yourself with the SOTA models....or read the corroborating literature... "General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Tasks" https://arxiv.org/abs/2604.11778 "...General365, a benchmark specifically designed to assess general reasoning in LLMs. By restricting background knowledge to a K-12 level, General365 explicitly decouples reasoning…
What proportion of the human population could answer the example from that paper? Question: Strangers A, B, C, D, and E line up from youngest on the left to oldest on the right. Their clothing colors and shoe colors all differ, and they come from five different regions. Known facts: 1. A is from Morocco. 2. D is five years older than B. 3. E is older than A. 4. C stands next to D. 5. A stands next to B. 6. The person…
This is a neat puzzle but you really don't need prolog. Spend a couple minutes thinking about it and you can solve this just fine.
Your question: It just means they have both green shoes and clothes, probably.
It's aimed to confuse a model with a bunch of information you-just-don't-really-need. The trick is to fish out which hints are more important than others. The order in which you apply them matters to make the puzzle easy vs. very hard. Which things give you the most amount of 'useful' information?
Call the people p1 to p5 (p5 oldest, p1 youngest).
Start with (15). This identifies the total age as 35+51. Now combine in (14), and you have 51 - 24 = 27 remaining years. This means the 35-yo is the oldest person, p5 = 35. Now add in the interesting thing that D = B+5. If you know that there is 27 years left to give to the two unknowns, B can't be 30 (or, D can't be p5). That means that D is 17 and B is 12 or D is 12 and B is 7.
This leaves three possible answers for the list of ages. Two of these are 7, 12, 12, 20, 35, or 10,12,12,17,35. The third one involves the case where the people other than the 12-year-olds and the 35-year-old are 5 years apart, in which case they can only be 11 and 16.
Combining all the 'stands-next-to' information (including point 2 which means D is to the right of B), we have eight possible orders: (AB)E(CD) or (AB)(CD)E. The last set of four is the only valid possible one as we know that D is not the eldest, which means E is 35. If the order is BADCE then B=7, A&D are 12, C=20, E=35. If the order is ABCDE then A=10,B=12,C=12,D=17,E=35 There's also the possibility that the 27 is made up of B and A (BACDE) are 11 and 16, and you are left with B=11,A=12,C=12,D=16,E=35. There is the possibility of the order being ABEDC. However, this carries no real consequences other than the clothing and shoes needing to be shuffled later. There's still only one possible value for the age of D. I'll leave the explanation for that variant out of the rest of the solution, the logic is roughly the same just with the clothes and country all juggled up.
Now one thing that's kind of unclear though is point (15). "One Adult". You'd need the definition of 'adult' to be a bit more precise (varies by country!), and also what kind of English is used to know if this implies there are at least 2 adults. If it does, then there is only one answer: D is always 12, since you need the 20-year-old to exist to have more than one adult.
If not, then D could be either 12 or 16 or 17, so the answer (so far) is 12x16x17 = 204x16=1224.
Now let's deal with the clothing, country, and shoes. All we need is that A is from Morocco, and that the second person is from South Africa. You can't simultaneously be from both places, which means A can't be in the second position.
That also clears up the ambiguity whether the 17 year old is an adult here (apparently they are), so that leaves only one possible combination: A,B,C,D,E, being 10,12,12,17,35. The answer to the question is 17.
Bonus: There's some extra stuff about countries and clothes: we have to check if there exist a valid solution that will fit with our one test. You could assume that the puzzle has an answer, but sneakily: that answer could be zero. If there are no solutions, there are no possible ages C could have, and so the answer is zero (or 1,depending on how you interpret the math of multiplying the empty set) . Here's one order that's possible (there's more, I'll leave it as an exercise to find the others):
Person: Country, Clothes, Shoes A(10): Morocco, Unknown, Red, B(12): South Africa, White, Orange C(12): Chile, Blue, Yellow D(17): Unknown, Green, Green E(35): Vanuatu, Yellow, Teal
(6) is the only point that's maybe a bit debatable? It's not true since the person wearing the teal shoes IS from vanuatu.
If this isn't valid solution for (6)... then there are no solutions, and the answer is actually 0. I'd go with 17 though, it seems fine.
Re: What sort of maths are LLMs good at?
#167Re: What sort of maths are LLMs good at?
#168From my experience, you have to be good at math to trust an LLM to do the math.
That's been an interesting thing for me with Claude Code and working with non-technical coworkers. Often they ask me to write something that could be a simple script and takes maybe 3 prompts to get it right, and I find myself wondering why they can't just do it themselves. But then I have them asking me what a .md file is and how to open it, and they're unable to follow readme instructions to use commandline program…
true, basic tech literacy is also a real bottleneck.
Re: What sort of maths are LLMs good at?
#169From my experience, you have to be good at math to trust an LLM to do the math.
There is an old koan in the old hacker literature: "A novice was trying to fix a broken Lisp machine by turning the power off and on. "Knight[, one of the principle designers of the Lisp machine], seeing what the student was doing, spoke sternly: 'You cannot fix a machine by just power-cycling it with no understanding of what is going wrong.' "Knight turned the machine off and on. "The machine worked." I feel like AI…
Exactly. AI is an amplifier, not a substitute, if you multiply high domain knowledge by AI you get incredible leverage, but 0 knowledge times AI is still effectively 0.
Re: What sort of maths are LLMs good at?
#170Earlier quoted context omitted.
There is an old koan in the old hacker literature: "A novice was trying to fix a broken Lisp machine by turning the power off and on. "Knight[, one of the principle designers of the Lisp machine], seeing what the student was doing, spoke sternly: 'You cannot fix a machine by just power-cycling it with no understanding of what is going wrong.' "Knight turned the machine off and on. "The machine worked." I feel like AI…
Exactly. AI is an amplifier, not a substitute, if you multiply high domain knowledge by AI you get incredible leverage, but 0 knowledge times AI is still effectively 0.
[dead]