Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI language models are struggling to “get” math

spectrum.ieee.org

Post reply on HN
Print view View original

101–110 of 201 posts

Re: AI language models are struggling to “get” math

#101

Post by mjburgess »

pessimizer wrote: post #85
mjburgess wrote: post #71

Earlier quoted context omitted.

I think it's deeply depressing that AI has been sold as something even capable of modelling anything humans do; and quite depressing that this comment exists. "AI" is just taking `mean()` over our choice of encodings of our choice of measurements of our selection of things we've created. There is as much "alike humans" in patterns in tree bark. AI is an embarrassingly dumb procedure, incapable of the most basic homol…

I lack your confidence, and find it a bit religious. > The idea that `mean(424,34324,223123,3424,....)` even has any revelance to us at all is quite absurd. Most of what I say to anyone is exactly this. When I'm about to give anyone any information, I look back at all of the relevant past information that I can recall (through word and sensory association, not by logic, unless I have a recollection of an associated i…

You're an animal. You've no idea what you do, and you're using machines as a model. Likewise, in the 16th C. it was brass cogs; and in anchient greece, air/fire/etc.

You're no more made of clay & god's breath, as you are sand and electricy.

You're an oozing, growing, malluable organic organism being physiologically dynamically shaped by your sensory-motor oozing. You're a mystery to yourself, and these self-reports, heavily coloured by the in-vogue tech are not science, they're pseudoscience.

If you want to study how animals work, you'd need to study that. Not these impoverished metaphors that mystify both machines and men. No machine has ever acquired a concept through sensory-motor action, nor used one to imagine, nor thereby planned its actions. No machine is ever at play, nor has grown its muscles to be better at-play. No machine has, therefore, learned to play the piano. No machine has thought about food, because no machine has been hungry; no machine has cared, nor been motivated to care by a harsh environment.

An inorganic mechanism is nothing at all like an animal, and an algorithm over a discrete sequence of numbers with electronic semantics, is nothing like tissue development.

What you are doing is not something you can introspect. And you arent really doing that. Rather, you've learned a "way of speaking" about machine action and are back-projecting that onto yourself. In this way, you're obliterating 95% of the things you are.

Re: AI language models are struggling to “get” math

#103

Post by macrolocal »

phillipharr1s wrote: post #93
vladf wrote: post #61

Earlier quoted context omitted.

Have you heard the piano continuations of AudioLM? https://google-research.github.io/seanet/audiolm/examples/

Pretty sure the first continuation is a famous piece with a few notes messed up. Can't remember the name. Honestly it only sounds marginally better than the old markov chain continuations.

Yep, Moonlight Sonata (mov. 3) no less. Talk about over-fitting!

Re: AI language models are struggling to “get” math

#104

Post by mjburgess »

woah wrote: post #84
mjburgess wrote: post #27

How much of this is just "AI is bad at everything", but in the math case, it's easier for the lay person to tell . It's all just passable garbled nonesense that the reader (goes to lengths) to interept based on their prior knowledge, which is not expressed in the syntax of what these systems output. In the case of mathematics, we're far less willing to "BS away" the interpretive failures. But if we were equally deman…

Have you ever used Github CoPilot? It does a lot of useful work, automating away rote typing in programming. Have you tried Dall-E or Stable Diffusion? They make good looking images. This comment seems completely unmoored from where the state of the art is right now.

sure, but co-pilot is mostly just copying code (see, for example, the issue with it producing quake source code).

If you think of AI as a dial from sample(data) to mean(data), then as the dial is turned towards the mean() you get more "generic" results, but also more garbled ones.

Copilot is more like a search engine, having turned the dial more towards sample().

The real invention of the NN is simply to provide that dial in a trainable way.

The only change to the "state of the art" is the size of the weights, and how long they take to train. This "advancement" is no more impressive than google indexing more webpages.

There has been no step-change advancement in AI in, perhaps, 50 years. All we see today is a product of hardware, in GPU/CPUs able to compress TBs of data into c. 300GB of weights. And likewise, the internet to provide it and SSDs to hold it.

The "magic" of AI is no more the magic of wikipida, here: copilot is good only because million+ programmers made github good.

It's still little more than a fancy search.

Re: AI language models are struggling to “get” math

#105

Post by gamegoblin »

the_af wrote: post #75
sharemywin wrote: post #3

Earlier quoted context omitted.

computers already do math. language models just need to translate problems into code of some kind that can be run to get the answer. executive function/planning is probably the biggest problem at this point for ai.

> language models just need to translate problems into code of some kind that can be run to get the answer A huge "just"! Isn't this the magic step? Translating ambiguous symbols to meaning and combining them in meaningful ways is a big deal which, apparently, these AI models cannot do. They can just parrot things.

It’s already being done and will only get better: https://twitter.com/sergeykarayev/status/1569377881440276481

Re: AI language models are struggling to “get” math

#106

Post by Marazan »

woah wrote: post #84
mjburgess wrote: post #27

How much of this is just "AI is bad at everything", but in the math case, it's easier for the lay person to tell . It's all just passable garbled nonesense that the reader (goes to lengths) to interept based on their prior knowledge, which is not expressed in the syntax of what these systems output. In the case of mathematics, we're far less willing to "BS away" the interpretive failures. But if we were equally deman…

Have you ever used Github CoPilot? It does a lot of useful work, automating away rote typing in programming. Have you tried Dall-E or Stable Diffusion? They make good looking images. This comment seems completely unmoored from where the state of the art is right now.

Dall-E produces good looking images within certain parameters.

When you are in its bounds it seems magical, once you go outside it seems like a weak joke.

And many of the reasons it is bad outside its sweet spot are fundamental to how it works not a flaw that can be iterated away.

Re: AI language models are struggling to “get” math

#107

Post by mlajtos »

yshrestha wrote: post #9

Language models can generate a Python function that does the math perfectly. I bet you would get better results if you tweaked the prompt to say "Generate a Python program that solves X math problem" and then just ran the resulting Python script. It does not need to be AGI to be useful.

Exactly, we need computer-equipped neural nets. Models need to use traditional UIs (including programming languages) and then we can talk about how to stop them. :)

Re: AI language models are struggling to “get” math

#108

Post by sebastialonso »

mgraczyk wrote: post #82
pessimizer wrote: post #76

That's because they're not modelling anything. The shocking thing about current AI models is that just sort of repeating and copying from memory what you've heard and seen gets you 97% of the way to imitating a person.* They still need to generate actual models somewhere to create consistency; so many generated images with one eye completely different from the other, or three arms, or fingers that grow into their cel…

This is factually wrong, both in terms of quantity and quality. Current AI models are not "just sort of repeating and copying from memory". This is just an incorrect characterization of how they work and how they perform. AI skeptics often say things like this then backpedal with something like "Well they aren't really repeating what they heard, but their generative model is just a slightly more sophisticated version…

can you actually share what "current AI models" are then? Not trying to be rude, but you just said "na ah" and then refused to argument any position.

Re: AI language models are struggling to “get” math

#109

Post by woah »

mjburgess wrote: post #104
woah wrote: post #84

Earlier quoted context omitted.

Have you ever used Github CoPilot? It does a lot of useful work, automating away rote typing in programming. Have you tried Dall-E or Stable Diffusion? They make good looking images. This comment seems completely unmoored from where the state of the art is right now.

sure, but co-pilot is mostly just copying code (see, for example, the issue with it producing quake source code). If you think of AI as a dial from sample(data) to mean(data), then as the dial is turned towards the mean() you get more "generic" results, but also more garbled ones. Copilot is more like a search engine, having turned the dial more towards sample(). The real invention of the NN is simply to provide that…

> It's all just passable garbled nonesense that the reader (goes to lengths) to interept based on their prior knowledge, which is not expressed in the syntax of what these systems output.

> It's still little more than a fancy search.

I feel like the goalposts have been moved between your two comments. CoPilot is obviously not producing garbled nonsense, and it's also not just printing the top result from StackOverflow. It is producing code that references my variables, does the right thing 50% of the time, and usually compiles.

One of the nice little things is error messages- when I type `if (!foo) { throw ... ` CoPilot is able to complete a nicely formatted and descriptive error message from its understanding of my code. It's not garbled nonsense, and it's not just a search engine.

Does AI deserve the hype it sometimes gets? Not yet. But I think you're going to have to start digging a little deeper for your commentary.

Re: AI language models are struggling to “get” math

#110

Post by holub008 »

p1esk wrote: post #100
vladf wrote: post #96

Earlier quoted context omitted.

Isn’t that as good as it gets? The whole point of the continuations is that given a short leading prompt from a real piece that it should continue it realistically. It didn’t get to train on the test set, if that’s what you’re implying, and I find it hard to believe the assertion that continuations are copies of the train set (if that’s your claim).

It definitely copied a piece of Moonlight sonata in the last 7 seconds of the first continuation sample: https://youtu.be/4Tr0otuiQuU?t=516

Interestingly, the original piece is a later Beethoven Sonata, Op. 31 No. 3. The model has its styles down! https://youtu.be/P-Q5aBAw-T4?t=78
Post reply on HN