Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI language models are struggling to “get” math

spectrum.ieee.org

Post reply on HN
Print view View original

151–160 of 201 posts

Re: AI language models are struggling to “get” math

#151

Post by moonchrome »

woah wrote: post #84
mjburgess wrote: post #27

How much of this is just "AI is bad at everything", but in the math case, it's easier for the lay person to tell . It's all just passable garbled nonesense that the reader (goes to lengths) to interept based on their prior knowledge, which is not expressed in the syntax of what these systems output. In the case of mathematics, we're far less willing to "BS away" the interpretive failures. But if we were equally deman…

Have you ever used Github CoPilot? It does a lot of useful work, automating away rote typing in programming. Have you tried Dall-E or Stable Diffusion? They make good looking images. This comment seems completely unmoored from where the state of the art is right now.

I haven't used copilot because I'm not sure I'm allowed, but I'll try it on a personal project eventually.

I'm hoping it's not as bad as Dall-E and Stable Diffusion - I've tried to use those to generate some generic product looking stock photos for a demo and it's spectacularly bad. The only context I see it get praised is fantasy style art - and that is visually appealing nonsense by definition.

If the code generated by copilot has the same "looks convincing but is fundamentally flawed" quality then it sounds like an insidious bug generator.

Re: AI language models are struggling to “get” math

#153

Post by NateEag »

p1esk wrote: post #99
dwringer wrote: post #88

Earlier quoted context omitted.

MIDI is extraordinarily expressive and is likely used to sequence a large majority of music produced within the last three decades. A lot of the instruments you hear are synthesizers or samplers running directly from MIDI. There is a lot more to what MIDI can do, and is used for, than the conception most people have from "canyon.mid" or old website background music. If an AI can do MIDI just fine then it's an extreme…

If an AI can do MIDI just fine then it's an extremely small leap to doing audio just fine. Unfortunately this is not true. It takes a huge amount of human effort to make MIDI encoded music sound good. The difference between MIDI and raw audio music generation is the same as the difference between drawing a cartoon and producing a photograph. To clarify, yes MIDI can be expressive, but what's being generated when peop…

thank you for clarifying this.

As a clasically-trained pianist who then got into electronica and synthesis, it was mind blowing to me that people could wrangle expression and phrasing from a MIDI sequencer.

Re: AI language models are struggling to “get” math

#154

Post by ma2rten »

qualudeheart wrote: post #149

The title is contradicted by the contents of the article. Minerva’s 78% accuracy is a miraculous result considering the prior 7% SOTA.

And almost none of the commenters seem to have read the article, so everyone is saying how predictable this was. Quite bizarre.

Re: AI language models are struggling to “get” math

#155

Post by qualudeheart »

CommieBobDole wrote: post #23

Also Excel is terrible at encoding MP3s. It's a language model; why would we expect it do math or try to somehow shoehorn math into the model? Do the language centers of our brain do math? If something approximating AGI is going to happen, it's going to be a lot of models tied together with an executive function to recognize and send things to the area that's good at working with them.

Language models aren’t even terrible at math. The Minerva paper provides a devastating counterexample. It will soon be replaced by more powerful linguistic-mathematical systems. Within the next twelve months we may well see 100% performance on all major benchmarks.

Re: AI language models are struggling to “get” math

#156

Post by qualudeheart »

ma2rten wrote: post #154
qualudeheart wrote: post #149

The title is contradicted by the contents of the article. Minerva’s 78% accuracy is a miraculous result considering the prior 7% SOTA.

And almost none of the commenters seem to have read the article, so everyone is saying how predictable this was. Quite bizarre.

It’s wishful thinking that I myself have once fallen for. I don’t trust our society to transition to a world with powerful machine intelligence safely, so would prefer a world in which ML progresses at a glacier’s pace.

Re: AI language models are struggling to “get” math

#157

Post by kazinator »

mjburgess wrote: post #27

How much of this is just "AI is bad at everything", but in the math case, it's easier for the lay person to tell . It's all just passable garbled nonesense that the reader (goes to lengths) to interept based on their prior knowledge, which is not expressed in the syntax of what these systems output. In the case of mathematics, we're far less willing to "BS away" the interpretive failures. But if we were equally deman…

> How much of this is just "AI is bad at everything"

"AI Language Models" are not touted as some general AI that is smart at everything, like a clever person with multiple intellectual skills integrated into one.

AI language models are for modeling language, not for math problem solving, or anything else. People good at language aren't always good at math.

DeepL produces very good, correct translations for "Alice has five more balls than Bob, who has two balls after he gives four to Charlie. How many balls does Alice have?" into numerous languages, even though it doesn't offer a solution.

I have little doubt that an AI system could be trained to translate word problems like this into systems of equations, which could be dumped into a some decades-old CAS to obtain a solution, which the AI could map back into the verbal domain through the identities between the math variables like x and Alice's apples.

"Hey look, that human who is supposedly good at math can't produce a painting of the Grand Canyon in the style of Monet, even if given eight months to do it, and is easily defeated in chess."

Re: AI language models are struggling to “get” math

#158

Post by mgraczyk »

smeagull wrote: post #146
mgraczyk wrote: post #91

Earlier quoted context omitted.

I'm disagreeing with the language you are using to characterize models. "copying from memory" implies that there is something being copied, and a memory that you are copying it from. I am pointing out that LLMs do not do this. It's not how they work. If you polled 1M random English speakers randomly and asked them whether or not a system that "just sort of repeating and copying from memory" could produce completely n…

The memory is the model. Having high probability to a particular area in latent space is copying it from the memory of the training data. You're a massive pedant, and this style of comment is a large reason HN comment sections are a pain to read.

What is the difference between this and describing a human brain the same way? The brain is the model, you are "just" copying things from the memory of your brain to words that you speak or write?

I don't think it's pedantic to say that an argument is wrong because it's making an incorrect claim. The claim here is that there is something different or missing between a true "model" and LLMs, and that missing thing has something to do with "copying". But that's not true, the missing thing is the complexity of the table, or the size of the table. The fact that it's copying in some incredibly abstract sense doesn't matter.

Re: AI language models are struggling to “get” math

#159

Post by intelVISA »

pessimizer wrote: post #76

That's because they're not modelling anything. The shocking thing about current AI models is that just sort of repeating and copying from memory what you've heard and seen gets you 97% of the way to imitating a person.* They still need to generate actual models somewhere to create consistency; so many generated images with one eye completely different from the other, or three arms, or fingers that grow into their cel…

"repeating and copying from memory what you've heard and seen gets you 97% of the way to imitating a person."

I'm in this hot take and I don't like it..

Re: AI language models are struggling to “get” math

#160

Post by zmgsabst »

NateEag wrote: post #150
zmgsabst wrote: post #121

Earlier quoted context omitted.

Automated theorem proving is the same problem as “complete and label the diagram”, which image generation is okay at. Work in progress for sure, though.

> Automated theorem proving is the same problem as “complete and label the diagram”, which image generation is okay at. How so? I'm no mathematician, but I don't see how these problem types are equivalent. Could you elaborate?

Sure — the connecting topic is topos theory.

For a type theory we might want to reason about, there’s a diagram (in category theory) which represents the same semantic content. These diagrams turn out to have recurring and common structures.

You can represent those diagrams as adjacency matrices, where those structures have a particular “shape” in the entries. Which if you squint hard looks like an image completion problem, ie, finding missing part of the matrix which represents a proof.

Post reply on HN