Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI language models are struggling to “get” math

spectrum.ieee.org

Post reply on HN
Print view View original

61–70 of 201 posts

Re: AI language models are struggling to “get” math

#61

Post by vladf »

dimmuborgir wrote: post #39
mjburgess wrote: post #27

How much of this is just "AI is bad at everything", but in the math case, it's easier for the lay person to tell . It's all just passable garbled nonesense that the reader (goes to lengths) to interept based on their prior knowledge, which is not expressed in the syntax of what these systems output. In the case of mathematics, we're far less willing to "BS away" the interpretive failures. But if we were equally deman…

AI is bad at music also. Even the state of the art transformer models can't produce more than a few seconds of coherent melodic phrases.

Have you heard the piano continuations of AudioLM?

https://google-research.github.io/seanet/audiolm/examples/

Re: AI language models are struggling to “get” math

#63

Post by [deleted] »

dimmuborgir wrote: post #39
mjburgess wrote: post #27

How much of this is just "AI is bad at everything", but in the math case, it's easier for the lay person to tell . It's all just passable garbled nonesense that the reader (goes to lengths) to interept based on their prior knowledge, which is not expressed in the syntax of what these systems output. In the case of mathematics, we're far less willing to "BS away" the interpretive failures. But if we were equally deman…

AI is bad at music also. Even the state of the art transformer models can't produce more than a few seconds of coherent melodic phrases.

[deleted]

Re: AI language models are struggling to “get” math

#64

Post by aaroninsf »

dimmuborgir wrote: post #39
mjburgess wrote: post #27

How much of this is just "AI is bad at everything", but in the math case, it's easier for the lay person to tell . It's all just passable garbled nonesense that the reader (goes to lengths) to interept based on their prior knowledge, which is not expressed in the syntax of what these systems output. In the case of mathematics, we're far less willing to "BS away" the interpretive failures. But if we were equally deman…

AI is bad at music also. Even the state of the art transformer models can't produce more than a few seconds of coherent melodic phrases.

AI can be quite good at music,

but yes there is not yet at on-demand button rendering from a text prompt of bitstreams encoding composed performed and mastered music.

Re: AI language models are struggling to “get” math

#65

Post by alan-crowe »

Buttons840 wrote: post #4

Are there any general purpose models that are good at learning math? I mainly know basic feed-forward neural nets, but I don't think they do well outside their training region. Math, of course, has an infinite training region.

I attempted to create a general purpose model for the exact version of the "what comes next problem." It enumerated primitive recursive functions, trying them out as it went. The limitation to primitive recursive functions was convenient because they always terminate. I didn't have to filter out the functions that ran for too long. (or do I?)

The enumeration inherently includes functions of several variables, so I wasn't restricted to examples such as 1->1, 2->4, 3->9, 4->16 etc.

I could try it out on examples such as (1,2)->3 (2,1)->3 (0,2)->2, etc. Perhaps with enough it would "learn to add" = find a primitive recursive function that did addition.

I got as far as finding the first problem. The enumeration technique that I used was effectively doing a tree recursion, like that function for computing Fibonacci numbers that bogs down because Fib(10) is computing Fib(5) lots of times. I had a lot of numbers that coded for the identity function, lots of numbers that coded for the first few functions, making the whole thing bog down, trying the same few functions over and over under different numerical disguises.

I thought that I could see my way to fixing this first problem. Have some way of recognizing numbers that give forms that give the same function. I guessed that I could approximate this by saying that if two functions give the same value on a variety of arguments they are probably the same. Then I parameterise this criterion and tune. That opens the way to creating a consolidated enumeration, analogous to fixing the tree recursive fibonacci function by memoization, except trickier.

But my health is poor and I ran out of energy.

Also, I have a guess for the second problem. What happens if I fix the first problem and my enumeration reaches decently complicated primitive recursive functions. While they will all terminate, some might run for far too long, causing the process to bog down. Rejecting them on the basis of limiting the run time might work well. We are happy to only learn reasonably effect functions for doing maths.

It is a fun idea and I encourage others to have a go.

Re: AI language models are struggling to “get” math

#66

Post by zackmorris »

sharemywin wrote: post #3
PaulHoule wrote: post #2

Ashby strikes again. Current sequence models don't have the right structures to represent math. Even if they use floating point internally, they can't really float the point because the nonlinearity in the model has a certain scale. A system that processes language can take advantage of the human desire for closure https://www.eurogamer.net/blood-in-the-gutter to fool people into thinking it is more capable than it r…

computers already do math. language models just need to translate problems into code of some kind that can be run to get the answer. executive function/planning is probably the biggest problem at this point for ai.

That's interesting, I hadn't made the connection between executive function and intelligence.

I went through a burnout in 2019 that felt like having a stroke. My brain finally reached such a level of negative reinforcement after years of failure that it wouldn't let me work anymore. I'd go to do very simple tasks, everything from brushing my teath to writing a TODO list, and it was like the part of my brain that performed those tasks wasn't there anymore. Or at least, it no longer obeyed if it perceived a potential reward involved. It was like my motivation got reversed. I had to relearn how to do everything, despite knowing that no reward might come for a very long time, which took at least 6 months before I began recovering. The closest answer I have is that my brain healed through faith.

I only bring it up because executive function may be associated with a subjective experience of meaning. If there's truly no point to anything, then it's hard to summon the motivation to string together a sequence of AI tasks into something more like AGI.

I guess that's another way of saying that nihilism could be the final hurdle for AGI to overcome. It's like the human philosophical question of why there's something instead of nothing. Or why angels would choose to be incarnate on Earth to experience a life of suffering when it's so much easier to remain dissociated.

Re: AI language models are struggling to “get” math

#67

Post by aaroninsf »

causi wrote: post #48
CactusOnFire wrote: post #41

Earlier quoted context omitted.

AI is bad at Audio. AI can do MIDI fine.

Which is a real shame. AI-powered restoration of poor-quality audio would be highly useful.

That particular niche has had some pretty amazing successes already. It's coming.

We can't produce arbitrary media streams with many "stack layers" of meaning and detail yet, but we can do a lot of specific instrumental transformations...

Vaguely relevant: https://koe.ai/recast/

Re: AI language models are struggling to “get” math

#69

Post by IshKebab »

JacobiX wrote: post #5
sharemywin wrote: post #3

Earlier quoted context omitted.

computers already do math. language models just need to translate problems into code of some kind that can be run to get the answer. executive function/planning is probably the biggest problem at this point for ai.

I'm not so sure about that. Of course computers can do arithmetic operations, but this is not the same as solving math problems, proving theorems, etc. Even mathematical objects are approximated up to an approximation error in a computer (like a differentiable manifold or a real number).

Computers are much much better at all that stuff than almost everyone too. Try asking Wolfram Alpha to solve something. Computers have gotten really good at proving things in the last couple of decades and formal verification methods are becoming increasingly popular.

I think sharemywin is probably on to something. It's going to be really hard for an AI to prove that e.g. x>0 && x+y 1 is unsatisfiable, but it's trivial for an SMT solver. On the other hand it probably isn't that much of a leap to make an AI that can feed that problem into an SMT solver.

Re: AI language models are struggling to “get” math

#70

Post by yeasurebut »

dimmuborgir wrote: post #59
yeasurebut wrote: post #55

Earlier quoted context omitted.

That’s what a musician does. They make short loops and loop them. This reads like someone who knows sheet music and theory but does not listen to music. It’s repetition of short phrases over and over. I’m not really sure what people expect of general AI trained on human generated outputs. It can’t make up anything anything “net new” only compose based upon what we feed it. I like to think AI is just showing us how si…

Those models are not trained on short loops. They are trained on whole songs just like image generation models are trained on whole images. And yet they struggle to repeat sections, modulate to a different key, create bridges, intros and outros. After a few seconds of hallucinating a melodic line they simply abandon the idea and migrate to another one. There is no global structure whatsoever.

Musicians don’t spit out an album in one sitting and they’re highly trained in theory. They get bored and tired of a process and take breaks. They come up with an album of loops composed together over time.

AIs state will forever be constrained to the limits of human cognition and behavior as that’s what it’s trained on.

I read published research all year. Circular reasoning. Tautology. It’s all over PhD thesis.

There’s no “global structure” to humanity. Relativity is a bitch.

Seeing the world through the vacuum of embedded inner monologue ignores the constraints of the physical one. It’s exhausting dealing with the mentality some clean room idea we imagine in a hammock can actually exist in a universe being ripped asunder by entropy.

It’s living in memory of what we were sold; some ideal state. Very akin to religious and nation state idealism.

Post reply on HN