Mathematical exploration and discovery at scale

AlphaEvolve isn't an LLM - it's an evolutionary coding agent that uses an LLM for code generation.

https://deepmind.google/blog/alphaevolve-a-gemini-powered-co...

This is part of Google/DeepMind's "Alpha" branding (AlphaGo, AlphaZero, AlphaFold) of bespoke machine learning solutions to tough problems.

It sounds like AlphaEvolve might do well on Chollet's ARC-AGI test, where this sort of program synthesis seems to be the most successful approach.

I find Tao's use of "extremize" vs "maximize" a bit jarring - maybe this is a more normal term in mathematics?

> finding geometric constructions

Finding optimal geometric constructions. Every problem is an optimisation because AlphaEvolve is an optimiser.

> This is not correct. The evaluation function is arbitrary.

You say this and then show details of how the score is calculated. AlphaEvolve needs a number to optimise, because it is optimiser. It can't optimise true/false.

> The feedback that you've constructed a graph of size K with some property does not tell you what you need to do to construct a graph of size K + M with the same property.

The feedback that you've constructed a graph of size K tell you that you've constructed a bigger graph than a competing solution that only constructed a graph of size K-1 and are therefore a more promising starting point for the next round of mutation

If you're trying to solve a "does there exist an X" problem, the information that none of your candidates found (or was) an X doesn't give you any information about which of them you should retain for mutation in the next step. You need a problem of the form "find the best X" (or, rather "find a good X") and for that you need a score of how well you've done. If you can find a score that actually improves steadily until you find the thing you're trying to prove the existence of then great, but generally these problems are "find the best X" where it's easy to come up with a load of competing Xs.

> The novelty here is that the LLM replaces the human expert.

That's not the claim at all. Tao said the benefits are scaling, robustness and interpretability, not that it can be operated by someone who doesn't know what they're doing.

Read https://www.argmin.net/p/lore-laundering-machines

Given time, we may find out that the solutions in this paper were also in the literature, as was the case in the anecdotes from the linked article :)

When I see announcements that say "we used a language model for X, and got novel results!", I play a little game where I identify the actual function of the language model in the system, and then replace it with something actually suited for that task. Here, the language model is used as the mutation / crossover component of a search through the space of computer programs.

What you really want here is represent the programs using an information-dense scheme, endowed with a pseudoquasimetric such that semantically-similar programs are nearby (and vice versa); then explore the vicinity of successful candidates. Ordinary compression algorithms satisfy "information-dense", but the metrics they admit aren't that great. Something that does work pretty well is embedding the programs into the kind of high-dimensional vector space you get out of a predictive text model: there may be lots of non-programs in the space, but (for a high-quality model) those are mostly far away from the programs, so exploring the neighbourhood of programs won't encounter them often. Because I'm well aware of the flaws of such embeddings, I'd add some kind of token-level fuzzing to the output, biased to avoid obvious syntax errors: that usually won't move the embedding much, but will occasionally jump further (in vector space) than the system would otherwise search.

So, an appropriate replacement for this generative language model would be some kind of… generative language model. Which is why I'm impressed by this paper.

There are enough other contributions in this paper that slotting a bog-standard genetic algorithm over program source in place of the language model could achieve comparable results; but I wouldn't expect it to be nearly as effective in each generation. If the language model is a particularly expensive part of the runtime (as the paper suggests might be the case), then I expect it's worth trying to replace it with a cruder-but-cheaper bias function; but otherwise, you'd need something more sophisticated to beat it.

(P.S.: props for trying to bring this back on-topic, but this subthread was merely about AI hype, not actually about the paper.)

Edit: Just read §3.2 of the paper. The empirical observations match the theory I've described here.

> not that it can be operated by someone who doesn't know what they're doing.

It's operated by an LLM, not a human. There is no human in the loop.