Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mathematical exploration and discovery at scale

terrytao.wordpress.com

Post reply on HN
Print view View original

21–30 of 135 posts

Re: Mathematical exploration and discovery at scale

#21

Post by looobay »

mariusor wrote: post #19
nl wrote: post #18

Hopefully this will finally stop the continuing claims[1] that LLMs can only solve problems they have seen before! If you listen carefully to the people who build LLMs it is clear that post-training RL forces them to develop a world-model that goes well beyond a "fancy Markov chain" that some seem to believe. Next step is building similar capabilities on top of models like Genie 3[2] [1] eg https://news.ycombinator.c…

For the less mathematically inclined of us, what is in that discussion that qualifies as a problem that has not been seen before? (I don't mean this combatively, I'd like to have a more mundane explanation)

It means something that is too out-of-data. For example if you try to make an LLM write a program in a strange or very new language it will struggle in non-trivial tasks.

Re: Mathematical exploration and discovery at scale

#22

Post by topaz0 »

nl wrote: post #18

Hopefully this will finally stop the continuing claims[1] that LLMs can only solve problems they have seen before! If you listen carefully to the people who build LLMs it is clear that post-training RL forces them to develop a world-model that goes well beyond a "fancy Markov chain" that some seem to believe. Next step is building similar capabilities on top of models like Genie 3[2] [1] eg https://news.ycombinator.c…

I think it's disingenuous to characterize these solutions as "LLMs solving problems", given the dependence on a hefty secondary apparatus to choose optimal solutions from the LLM proposals. And an important point here is that this tool does not produce any optimality proofs, so even if they do find the optimal result, you may not be any closer to showing that that's the case.

Re: Mathematical exploration and discovery at scale

#23

Post by griffzhowl »

khafra wrote: post #9
nsoonhui wrote: post #8

Earlier quoted context omitted.

>> If the LLM fucks up, that branch is cut. Can you explain more on this? How on earth are we supposed to know LLM is hallucinating?

Math is a verifiable domain. Translate a proof into Lean and you can check it in a non-hallucination-vulnerable way.

But that's not what they're doing here. They're comparing Alphaevolve's outputs numerically against a scoring function

Re: Mathematical exploration and discovery at scale

#24

Post by mariusor »

looobay wrote: post #21
mariusor wrote: post #19

Earlier quoted context omitted.

For the less mathematically inclined of us, what is in that discussion that qualifies as a problem that has not been seen before? (I don't mean this combatively, I'd like to have a more mundane explanation)

It means something that is too out-of-data. For example if you try to make an LLM write a program in a strange or very new language it will struggle in non-trivial tasks.

I understand what "a new problem for an LLM is", my question is about what in the math discussion qualifies as a one.

I see references to "improvements", "optimizing" and what I would describe as "iterating over existing solutions" work, not something that's "new". But as I'm not well versed into maths I was hoping that someone that considers the thread as definite proof for that, like parent seems to be, is capable of offering a dumbed down explanation for the five year olds among us. :)

Re: Mathematical exploration and discovery at scale

#25

Post by swannodette »

nl wrote: post #18

Hopefully this will finally stop the continuing claims[1] that LLMs can only solve problems they have seen before! If you listen carefully to the people who build LLMs it is clear that post-training RL forces them to develop a world-model that goes well beyond a "fancy Markov chain" that some seem to believe. Next step is building similar capabilities on top of models like Genie 3[2] [1] eg https://news.ycombinator.c…

I don't see how anything about what's presented here that refutes such claims. This mostly confirms that LLM based approaches need some serious baby-sitting from experts and those experts can derive some value from them but generally with non-trivial levels of effort and non-LLM supported thinking.

Re: Mathematical exploration and discovery at scale

#26

Post by ineedasername »

topaz0 wrote: post #22
nl wrote: post #18

Hopefully this will finally stop the continuing claims[1] that LLMs can only solve problems they have seen before! If you listen carefully to the people who build LLMs it is clear that post-training RL forces them to develop a world-model that goes well beyond a "fancy Markov chain" that some seem to believe. Next step is building similar capabilities on top of models like Genie 3[2] [1] eg https://news.ycombinator.c…

I think it's disingenuous to characterize these solutions as "LLMs solving problems", given the dependence on a hefty secondary apparatus to choose optimal solutions from the LLM proposals. And an important point here is that this tool does not produce any optimality proofs, so even if they do find the optimal result, you may not be any closer to showing that that's the case.

Well, there's the goal posts moved and a Scotsman denied. It's got an infrastructure in which it operates and "didn't show its work" so it takes an F in maths.

Re: Mathematical exploration and discovery at scale

#28

Post by dpflan »

swannodette wrote: post #25
nl wrote: post #18

Hopefully this will finally stop the continuing claims[1] that LLMs can only solve problems they have seen before! If you listen carefully to the people who build LLMs it is clear that post-training RL forces them to develop a world-model that goes well beyond a "fancy Markov chain" that some seem to believe. Next step is building similar capabilities on top of models like Genie 3[2] [1] eg https://news.ycombinator.c…

I don't see how anything about what's presented here that refutes such claims. This mostly confirms that LLM based approaches need some serious baby-sitting from experts and those experts can derive some value from them but generally with non-trivial levels of effort and non-LLM supported thinking.

Yes, applied research has yielded the modern expert system, which is really useful to experts who know what they are doing.

Re: Mathematical exploration and discovery at scale

#29

Post by kzz102 »

It's really tiring that LLM fans will claim every progress as breakthrough and go into fantasy mode on what they can do afterwards.

This is a really good example of how to use the current capabilities of LLM to help research. The gist is that they turned math problems into problems for coding agents. This uses the current capabilities of LLM very well and should find more uses in other fields. I suspect the Alpha evolve system probably also has improvements over existing agents as well. AI is making steady and impressive process every year. But it's not helpful for either the proponents or the skeptics to exaggerate their capabilities.

Post reply on HN