Can AI do maths yet? Thoughts from a mathematician
xenaproject.wordpress.com
Can AI do maths yet? Thoughts from a mathematician
1–10 of 364 posts
Re: Can AI do maths yet? Thoughts from a mathematician
#2"once" the training data can do it, LLMs will be able to do it. and AI will be able to do math once it comes to check out the lights of our day and night. until then it'll probably wonder continuously and contiguously: "wtf! permanence! why?! how?! by my guts, it actually fucking works! why?! how?!"
Re: Can AI do maths yet? Thoughts from a mathematician
#3"once" the training data can do it, LLMs will be able to do it. and AI will be able to do math once it comes to check out the lights of our day and night. until then it'll probably wonder continuously and contiguously: "wtf! permanence! why?! how?! by my guts, it actually fucking works! why?! how?!"
I do think it is time to start questioning whether the utility of ai solely can be reduced to the quality of the training data.
This might be a dogma that needs to die.
Re: Can AI do maths yet? Thoughts from a mathematician
#4I may be wrong, but I think it a silly question. AI is basically auto-complete. It can do math to the extent you can find a solution via auto-complete based on an existing corpus of text.
Re: Can AI do maths yet? Thoughts from a mathematician
#5"once" the training data can do it, LLMs will be able to do it. and AI will be able to do math once it comes to check out the lights of our day and night. until then it'll probably wonder continuously and contiguously: "wtf! permanence! why?! how?! by my guts, it actually fucking works! why?! how?!"
I do think it is time to start questioning whether the utility of ai solely can be reduced to the quality of the training data. This might be a dogma that needs to die.
I tried. I don't have the time to formulate and scrutinise adequate arguments, though.
Do you? Anything anywhere you could point me to?
The algorithms live entirely off the training data. They consistently fail to "abduct" (inference) beyond any language-in/of-the-training-specific information.
Re: Can AI do maths yet? Thoughts from a mathematician
#6I may be wrong, but I think it a silly question. AI is basically auto-complete. It can do math to the extent you can find a solution via auto-complete based on an existing corpus of text.
You're underestimating the emergent behaviour of these LLM's. See for example what Terrence Tao thinks about o1:
Re: Can AI do maths yet? Thoughts from a mathematician
#7I may be wrong, but I think it a silly question. AI is basically auto-complete. It can do math to the extent you can find a solution via auto-complete based on an existing corpus of text.
Pretty sure this is out of date now
Re: Can AI do maths yet? Thoughts from a mathematician
#8Earlier quoted context omitted.
I do think it is time to start questioning whether the utility of ai solely can be reduced to the quality of the training data. This might be a dogma that needs to die.
I tried. I don't have the time to formulate and scrutinise adequate arguments, though. Do you? Anything anywhere you could point me to? The algorithms live entirely off the training data. They consistently fail to "abduct" (inference) beyond any language-in/of-the-training-specific information.
The best way to predict the next word is to accurately model the underlying system that is being described.
Re: Can AI do maths yet? Thoughts from a mathematician
#9I am fairly optimistic about LLMs as a human math -> theorem-prover translator, and as a fan of Idris I am glad that the AI community is investing in Lean. As the author shows, the answer to "Can AI be useful for automated mathematical work?" is clearly "yes."
But I am confident the answer to the question in the headline is "no, not for several decades." It's not just the underwhelming benchmark results discussed in the post, or the general concern about hard undergraduate math using different skillsets than ordinary research math. IMO the deeper problem still seems to be a basic gap where LLMs can seemingly do formal math at the level of a smart graduate student but fail at quantitative/geometric reasoning problems designed for fish. I suspect this holds for O3, based on one of the ARC problems it wasn't able to solve: https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_pr... (via https://www.interconnects.ai/p/openais-o3-the-2024-finale-of...) ANNs are simply not able to form abstractions, they can only imitate them via enormous amounts of data and compute. I would say there has been zero progress on "common sense" math in computers since the invention of Lisp: we are still faking it with expert systems, even if LLM expert systems are easier to build at scale with raw data.
It is the same old problem where an ANN can attain superhuman performance on level 1 of Breakout, but it has to be retrained for level 2. I am not convinced it makes sense to say AI can do math if AI doesn't understand what "four" means with the same depth as a rat, even if it can solve sophisticated modular arithmetic problems. In human terms, does it make sense to say a straightedge-and-compass AI understands Euclidean geometry if it's not capable of understanding the physical intuition behind Euclid's axioms? It makes more sense to say it's a brainless tool that helps with the tedium and drudgery of actually proving things in mathematics.