31–40 of 135 posts
Earlier quoted context omitted.
Math is a verifiable domain. Translate a proof into Lean and you can check it in a non-hallucination-vulnerable way.
But that's not what they're doing here. They're comparing Alphaevolve's outputs numerically against a scoring function
Earlier quoted context omitted.
>> If the LLM fucks up, that branch is cut. Can you explain more on this? How on earth are we supposed to know LLM is hallucinating?
Google's system is like any other optimizer, where you have a scoring function, and you keep altering the function's inputs to make the scoring function return a big number. The difference here is the function's inputs are code instead of numbers, which makes LLMs useful because LLMs are good at altering code. So the LLM will try different candidate solutions, then Google's system will keep working on the good ones a…
Hopefully this will finally stop the continuing claims[1] that LLMs can only solve problems they have seen before! If you listen carefully to the people who build LLMs it is clear that post-training RL forces them to develop a world-model that goes well beyond a "fancy Markov chain" that some seem to believe. Next step is building similar capabilities on top of models like Genie 3[2] [1] eg https://news.ycombinator.c…
Earlier quoted context omitted.
I think it's disingenuous to characterize these solutions as "LLMs solving problems", given the dependence on a hefty secondary apparatus to choose optimal solutions from the LLM proposals. And an important point here is that this tool does not produce any optimality proofs, so even if they do find the optimal result, you may not be any closer to showing that that's the case.
Well, there's the goal posts moved and a Scotsman denied. It's got an infrastructure in which it operates and "didn't show its work" so it takes an F in maths.
Isabelle/HOL has a tool called Sledgehammer, which is the hackiest hack that ever hacked[0], basically amounting to "run a load of provers in parallel, with as much munging as it takes". (Plumbing them together is a serious research contribution, which I'm not at all belittling.) I've yet to see ChatGPT achieve anything like what it's capable of.
[0]: https://lawrencecpaulson.github.io/2022/04/13/Sledgehammer.h...
Earlier quoted context omitted.
But that's not what they're doing here. They're comparing Alphaevolve's outputs numerically against a scoring function
They did also take some of the informal proofs and formalized them using AlphaProof, emitting Lean.
Hopefully this will finally stop the continuing claims[1] that LLMs can only solve problems they have seen before! If you listen carefully to the people who build LLMs it is clear that post-training RL forces them to develop a world-model that goes well beyond a "fancy Markov chain" that some seem to believe. Next step is building similar capabilities on top of models like Genie 3[2] [1] eg https://news.ycombinator.c…
The AlphaEvolve paper has been out since May. I don't think the people making these claims are necessarily primarily motivated by the accuracy of what they're saying.
Hopefully this will finally stop the continuing claims[1] that LLMs can only solve problems they have seen before! If you listen carefully to the people who build LLMs it is clear that post-training RL forces them to develop a world-model that goes well beyond a "fancy Markov chain" that some seem to believe. Next step is building similar capabilities on top of models like Genie 3[2] [1] eg https://news.ycombinator.c…
Hopefully this will finally stop the continuing claims[1] that LLMs can only solve problems they have seen before! If you listen carefully to the people who build LLMs it is clear that post-training RL forces them to develop a world-model that goes well beyond a "fancy Markov chain" that some seem to believe. Next step is building similar capabilities on top of models like Genie 3[2] [1] eg https://news.ycombinator.c…
Please read section 2 of the paper[1] cited in the blog post. LLMs are used as a mutation function in an evolutionary loop. LLMs are certainly an enabler, but IMO, evolutionary optimization is what deserves credit in this case. [1]: https://arxiv.org/abs/2511.02864
Hopefully this will finally stop the continuing claims[1] that LLMs can only solve problems they have seen before! If you listen carefully to the people who build LLMs it is clear that post-training RL forces them to develop a world-model that goes well beyond a "fancy Markov chain" that some seem to believe. Next step is building similar capabilities on top of models like Genie 3[2] [1] eg https://news.ycombinator.c…
Earlier quoted context omitted.
I think it's disingenuous to characterize these solutions as "LLMs solving problems", given the dependence on a hefty secondary apparatus to choose optimal solutions from the LLM proposals. And an important point here is that this tool does not produce any optimality proofs, so even if they do find the optimal result, you may not be any closer to showing that that's the case.
Well, there's the goal posts moved and a Scotsman denied. It's got an infrastructure in which it operates and "didn't show its work" so it takes an F in maths.