Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mathematical exploration and discovery at scale

terrytao.wordpress.com

Post reply on HN
Print view View original

81–90 of 135 posts

Re: Mathematical exploration and discovery at scale

#81

Post by youoy »

> AlphaEvolve did not perform equally well across different areas of mathematics. When testing the tool on analytic number theory problems, such as that of designing sieve weights for elementary approximations to the prime number theorem, it struggled to take advantage of the number theoretic structure in the problem, even when given suitable expert hints (although such hints have proven useful for other problems). This could potentially be a prompting issue on our end,

Very generous from Tao to say it can be a prompting issue. It always surprises me how easily it is for people to says that the problem is not the LLM, but them. With other types of ML/AI algorithms we dont see this. For example, after a failed attempt or lower score in a comparison table, no one writes "the following benchmark results may be wrong, and our proposed algorithm may not be the best. We may have messed up the hyperparameter tunning, initialization, train test split..."

Re: Mathematical exploration and discovery at scale

#82

Post by griffzhowl »

ants_everywhere wrote: post #44
piker wrote: post #3

That was dense but seemed nuanced. Anyone care to summarize for those of us who lack the mathematics nomenclature and context?

They put an LLM in a loop that mimics how people do real math, and it did research-level math. Like humans, it wasn't equally capable across all mathematical domains. The experiment was set up to mimic mathematicians who are excellent at proving inequalities, bounds, finding optimal solutions, etc. So more like Ramanujan and Erdős in their focus on a computationally-driven and problem-focused approach.

An LLM as a component of a tool, plus a team of research mathematicians, did research level math

Re: Mathematical exploration and discovery at scale

#83

Post by ants_everywhere »

omnicognate wrote: post #80
ants_everywhere wrote: post #77

Earlier quoted context omitted.

The objective function is the problem. The objective function is given in all problems. In "find an X such that Y" the objective function is "is this an X? is Y satisfied?". In induction you have P and n to ratchet up the proof by increasing n such that P(n) holds. Combining this reply with your previous one, it sounds like you're setting up a situation where the LLM can neither know the problem nor whether it's maki…

What's the objective function for the Langlands Program?

The Langlands Program is a research program not a single mathematical problem. It consists of many open conjectures with various levels of progress toward them. However, I think it's quite a victory that the goal posts have moved from "LLMs can't solve problems" to "LLMs probably can't solve all open conjectures in Langlands Program in one shot.

But as I said way up above, if you have the statement of any particular problem you can just use the function that evaluates proofs as your objective function. If you were to do this in Lean, for example, you'd get compiler output that contains information you can use to see if you're on the right track.

In addition to the output of the proof program you'd probably want to score sub-computations in the proof as metrics. E.g. if you want to show that a map has finite fibers, you may want to record the size of the fibers, or a max over the size. If you need to know an element is not contained in some Levi subgroup then you may want to record information about the relevant Levi decomposition. This mimics things that humans know to score their progress as they're doing computations.

Re: Mathematical exploration and discovery at scale

#84

Post by ants_everywhere »

griffzhowl wrote: post #82
ants_everywhere wrote: post #44

Earlier quoted context omitted.

They put an LLM in a loop that mimics how people do real math, and it did research-level math. Like humans, it wasn't equally capable across all mathematical domains. The experiment was set up to mimic mathematicians who are excellent at proving inequalities, bounds, finding optimal solutions, etc. So more like Ramanujan and Erdős in their focus on a computationally-driven and problem-focused approach.

An LLM as a component of a tool, plus a team of research mathematicians, did research level math

Yes mathematicians chose problems for the LLM to solve, and then the LLM solved them. That's how we know they were good open problems and that this is research level math.

Re: Mathematical exploration and discovery at scale

#85

Post by youoy »

ants_everywhere wrote: post #76
j2kun wrote: post #66

Earlier quoted context omitted.

> that mimics how people do real math Real people do not do math like AlphaEvolve...

What do you feel is fundamentally different about the feedback loop in AlphaEvolve compared to, say, Einstein and Grossman repeatedly running calculations until they found the right tensor setup for General Relativity? Or Euler filling his notebooks with computations? Or Ramanujan? Or Newton working out infinite series? Or Kepler, etc etc. They are all doing iterative search with feedback from a function that tells t…

The reward functions in the problems that they proposed alphaevolve are easy. The reward funtions of at least 50% of maths are not. You can say that validating if a proof is correct is a straightforward reward, but the size of interesting theorems over the space of all theorems is very small. And also what does "interesting" could even mean?

Re: Mathematical exploration and discovery at scale

#86

Post by dbacar »

smokel wrote: post #53
kzz102 wrote: post #29

It's really tiring that LLM fans will claim every progress as breakthrough and go into fantasy mode on what they can do afterwards. This is a really good example of how to use the current capabilities of LLM to help research. The gist is that they turned math problems into problems for coding agents. This uses the current capabilities of LLM very well and should find more uses in other fields. I suspect the Alpha evo…

One could say the same about these kinds of comments. If you don't like the content, simply don't read it? And to add something constructive: the timeframes for enjoying a hype cycle differ from person to person. If you are on top of things, it might be tiring, but there are still many people out there, who haven't made the connection between, in this case, LLMs and mathematics. Inspiring some people to work on this…

Did you read what she/he says?

Re: Mathematical exploration and discovery at scale

#87

Post by macleginn »

youoy wrote: post #81

> AlphaEvolve did not perform equally well across different areas of mathematics. When testing the tool on analytic number theory problems, such as that of designing sieve weights for elementary approximations to the prime number theorem, it struggled to take advantage of the number theoretic structure in the problem, even when given suitable expert hints (although such hints have proven useful for other problems). T…

Even without such acknowledgments it is hard to get past reviewers ("Have you tried more extensive hyperparameter tunning, other initializations and train test splits?"). These are essentially lab notes from an exploratory study, so (with absolutely no disrespect to the author) the setting is different.

Re: Mathematical exploration and discovery at scale

#89

Post by ajfkfkdjsfi »

smokel wrote: post #53
kzz102 wrote: post #29

It's really tiring that LLM fans will claim every progress as breakthrough and go into fantasy mode on what they can do afterwards. This is a really good example of how to use the current capabilities of LLM to help research. The gist is that they turned math problems into problems for coding agents. This uses the current capabilities of LLM very well and should find more uses in other fields. I suspect the Alpha evo…

One could say the same about these kinds of comments. If you don't like the content, simply don't read it? And to add something constructive: the timeframes for enjoying a hype cycle differ from person to person. If you are on top of things, it might be tiring, but there are still many people out there, who haven't made the connection between, in this case, LLMs and mathematics. Inspiring some people to work on this…

Ah yes, the bootlicker's desire of letting the bootlickers winout so there is only walls of bootlicking for any agnostic that happens across a post.

I'd rather dissent so others know dissent is a rational response.

Re: Mathematical exploration and discovery at scale

#90

Post by omnicognate »

ants_everywhere wrote: post #83
omnicognate wrote: post #80

Earlier quoted context omitted.

What's the objective function for the Langlands Program?

The Langlands Program is a research program not a single mathematical problem. It consists of many open conjectures with various levels of progress toward them. However, I think it's quite a victory that the goal posts have moved from "LLMs can't solve problems" to "LLMs probably can't solve all open conjectures in Langlands Program in one shot. But as I said way up above, if you have the statement of any particular…

If you want a simple, well defined problem that a child can understand the statement of, how about giving me the objective function for the Collatz Conjecture?
Post reply on HN