[deleted]
AI language models are struggling to “get” math
31–40 of 201 posts
Re: AI language models are struggling to “get” math
#32More generally, they struggle to get thing right. They’re great at grammatical confabulation, but when you need a correct answer, or a correct drug recommendation, ask an expert.
Re: AI language models are struggling to “get” math
#33Language models can generate a Python function that does the math perfectly. I bet you would get better results if you tweaked the prompt to say "Generate a Python program that solves X math problem" and then just ran the resulting Python script. It does not need to be AGI to be useful.
That could only generate constructivist [0] proofs, and there are many things done in modern maths which are not constructivist. Maybe a better approach would be to use Curry-Howard [1] correspondence to directly get proofs from generated programs
[0] https://en.wikipedia.org/wiki/Constructivism_(philosophy_of_...
[1] https://en.wikipedia.org/wiki/Curry%E2%80%93Howard_correspon...
Re: AI language models are struggling to “get” math
#34Are there any general purpose models that are good at learning math? I mainly know basic feed-forward neural nets, but I don't think they do well outside their training region. Math, of course, has an infinite training region.
There is "LODA", which uses genetic algorithms, that continuously mutates existing math programs until discovering something new. It uses OEIS as training data, around 350k known integer sequences, such as primes/fibonacci. Around 100k programs have been mined so far.
I'm a contributer to LODA.
LODA runs on CPU. It doesn't use GPU. If you have spare computer, then please consider contributing with the mining. Your contribution helps.
Re: AI language models are struggling to “get” math
#35Earlier quoted context omitted.
There has been big progress in automated theorem proving lately https://en.wikipedia.org/wiki/Automated_theorem_proving you just don't hear about it much because the technology is not so fashionable today. Also it is more clear what the limits are, I mean, Turing, Godel, Tarski and all of those apply to neural networks as well any other formal system but people mostly forget it. Knuth wrote a really fun volume of The…
> There has been big progress in automated theorem proving lately It doesn't seem like there has been much progress for anything but FOL?
The wiki article on automated theorem proving is quite bad as an overview of the active field; it's more a historical article about the mid to late 20th century. Most of the interesting things in automated reasoning have happened since the naughts, and that article kind of stops in the 90s
SMT solvers have gotten quite good over the past couple decades, there are tons of domain-specific tools (eg in software and hardware verification), tons of niche applied decidable or semi-decidable theories (eg various modal and description logics), a lot of progress on the proof assistant ("non-fully-automated theorem proving") paradigm, and so on.
Re: AI language models are struggling to “get” math
#36Language models can generate a Python function that does the math perfectly. I bet you would get better results if you tweaked the prompt to say "Generate a Python program that solves X math problem" and then just ran the resulting Python script. It does not need to be AGI to be useful.
you can also tell the model that it doesnt know how to do math, and it respects that
Re: AI language models are struggling to “get” math
#37The situation is actually much worse for science, or any moving field. This models are by design and necessity historical. So that if, for example, the FDA issues a drug approval overnight, The model camp follow sudden changes in a “reasoned” why.
Re: AI language models are struggling to “get” math
#38Earlier quoted context omitted.
I wonder how these language models would do if we tried to teach them maths the way schools do: Feed them explanations first, then endless sequences of toy problems, see which they got wrong and feed them corrected examples back in. I'm not at all surprised they don't do well at maths, because while there are maths texts online, I doubt there is enough material to give these models the same experience of repetition a…
Generating solved math problems is trivial, like making AlphaZero play itself in chess. Sparse Data is not the problem. Refusing to use it is.
I don't think it's so much a refusal, as that it's not been a sufficient priority for anyone before. As the article points out there are now a few training sets which includes math problems, and models which do well on them. But the remaining problems seems to be with basics which humans tends to learn to do consistently with a lot of repetition, and it'd be interesting to see those datasets extended to the very simple.
Re: AI language models are struggling to “get” math
#39How much of this is just "AI is bad at everything", but in the math case, it's easier for the lay person to tell . It's all just passable garbled nonesense that the reader (goes to lengths) to interept based on their prior knowledge, which is not expressed in the syntax of what these systems output. In the case of mathematics, we're far less willing to "BS away" the interpretive failures. But if we were equally deman…
AI is bad at music also. Even the state of the art transformer models can't produce more than a few seconds of coherent melodic phrases.
Re: AI language models are struggling to “get” math
#40Also Excel is terrible at encoding MP3s. It's a language model; why would we expect it do math or try to somehow shoehorn math into the model? Do the language centers of our brain do math? If something approximating AGI is going to happen, it's going to be a lot of models tied together with an executive function to recognize and send things to the area that's good at working with them.
Well, because we want rational language models. Something with a sense of truth. Math is not irrelevant—and I’m sure it’s a solvable problem with language models.
But if it's rational and has a sense of truth, then it's AGI. Which I don't think is impossible or even unattainable within a reasonable amount of time, but we're .001% of the way there, not 50% or 75%.
These models are fascinating, but the problem 'a lot of the things this model generates lack any semantic meaning' is inherent and likely insurmountable without connecting the model to other, far more complex models that haven't been built yet.
We are at the level where our models can consistently generate blocks of text with full sentences in them that make grammatical sense. Which is pretty cool.
But the next step is being able to consistently generate full sentences that make grammatical sense and usefully convey information. And while the current models do that a lot of the time, they don't do that all of the time because they don't and can't know the difference without essentially being a different thing. Because to do that consistently, we need an "understanding what things mean" model. Which is many orders of magnitude larger and more difficult than a text generator.