Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is AI reasoning right for the wrong reasons?

quantamagazine.org

Post reply on HN
Print view View original

101–110 of 268 posts

Re: Is AI reasoning right for the wrong reasons?

#101

Post by dinfinity »

zarzavat wrote: post #83

LLMs lack qualia, among other things. If I ask an LLM "what is an apple?" it tells me: > An apple is the edible fruit of the apple tree, scientifically known as Malus domestica. It is one of the world's most widely grown fruits and is eaten fresh or used in many foods and drinks. If I ask an LLM "what is a mundu fruit?" it tells me: > Mundu is a tropical fruit native to Southeast Asia, especially found in Indonesia,…

> LLMs lack qualia, among other things.

You don't know and can't know that. You cannot know that of any other being than yourself. If you understood what qualia are, you would know you can't know. It is the hard problem of consciousness and most easily explored through contemplating aphantasia and related concepts (how do blind people dream?).

> A mundu fruit is an abstract experience: text, images, only slightly more real than a fictional fruit. I'm aware that mundu fruit exist, just as the LLM is aware the apples exist, but that doesn't make them exist for me.

You're incorrectly elevating some senses over others (AI ingesting images is cognitively very comparable to seeing) and confusing 'having memories of an experience' with 'having experiences at all'. Tokens in the LLM context can also be seen as memories of an experience, albeit as a repeated "replay" of them rather than (longterm) internalization.

Re: Is AI reasoning right for the wrong reasons?

#102

Post by pegasus »

andy99 wrote: post #96
ilovecake1984 wrote: post #91

Earlier quoted context omitted.

This is a good take. People overwhelmingly fit a narrative to how they make decisions rather than make decisions according to a chain of reasoning.

That’s just conflating reasoning and “parallel reconstruction” - there is such thing as reasoning, and I agree it’s probably less common in decision making, but it’s fundamental to many tasks where we figure out a solution, like writing an algorithm, math proofs, etc.

Sure, but the same must be true for these models as well. They must be doing some form of reasoning, even if it's not formally correct and it relies instead on all kind of half-assed heuristics which only work inside certain unexamined boundaries, which leads to wrong conclusions when those boundaries are being trespassed, etc. And even if it's not the reasoning they report when asked. But all this can't be that different to how humans reason most of the time either.

I suspect humans have other ways that help with error correction and guiding the reasoning effort, but that's another story.

Re: Is AI reasoning right for the wrong reasons?

#103

Post by petesergeant »

andy99 wrote: post #13

Back in the day it was a bit of a cliche to bring up “clever Hans”, the horse that could do math, when talking about machine learning. He couldn’t do math but he read some cues from his handler of pick the write answers, the handler iirc wasn’t in on it. The point of the story was that classifiers can be right for the wrong reasons and almost inevitably are. At least there’s zero guarantee that the reason for making…

Aren't humans also classifiers? Where, precisely, is the dividing line between a sufficiently large model and an intelligence?

Re: Is AI reasoning right for the wrong reasons?

#104

Post by pixl97 »

andy99 wrote: post #13

Back in the day it was a bit of a cliche to bring up “clever Hans”, the horse that could do math, when talking about machine learning. He couldn’t do math but he read some cues from his handler of pick the write answers, the handler iirc wasn’t in on it. The point of the story was that classifiers can be right for the wrong reasons and almost inevitably are. At least there’s zero guarantee that the reason for making…

Reasoning is a term that's like the term intelligence in that it's highly complex and much less well defined than we'd like it to be, especially as we're trying to compare algorithms to human behaviors.

To begin to explore this we have to imagine a reasoning oracle, that when presented with a question that can be reasoned will immediately present a correct answer. Immediately from this definition we can see that humans are not perfect reasoning oracles. Also with definition of reasoning, any agent with generalized capabilities would quickly reach some kind of singularity or solution to the universe as much as is possible within this universe.

We've established that we are imperfect reasoners, the next question ends up being how do you measure reasoning capabilities? This comes with a second question of when does following an algorithm become reasoning? Animals for example have a large amount of instinctual reasoning as an inherit part of their architecture that self trains when presented with the correct environment. What does the gradient between low level reasoning and high level reasoning look like?

Coming back to humans, we know of countless example of where human reasoning was right for the wrong reasons. We also have seen many different paths to the same solution. For the egotistical it would be easy to say "well if you didn't do it they way I did, you are wrong", but that's not very scientific. Defining the topology of the problem is a more scientific way of looking at it, but quickly runs into P=NP and Kolmogorov complexity issues.

The last thing I'll touch on is why did human society begin to rapidly change after we started following the scientific method. Looking at the method in hindsight, it seems like a pattern that a reasoning agent should have been able to distil and follow around the time it reaches what we consider higher level reasoning. Therefor I'm highly resistant to consider human reasoning "real" reasoning versus any other possible path to reasoning. Human reasoning is simply what we are comfortable with. This also leads me to consider the worrying possibility there are far more efficient and faster means of reasoning that I as a human will be very uncomfortable with.

Re: Is AI reasoning right for the wrong reasons?

#105

Post by beering »

cmrdporcupine wrote: post #42
andrewla wrote: post #7

I'll admit that I find this discussion a bit navel-gazy. It has become a question of semantics not a question of actual functionality. The question has become "what do we mean when we use the word 'reasoning'" which is uninteresting. Dijkstra said[1] "... the question whether computers can think. The question is just as relevant and just as meaningful as the question whether submarines can swim." I don't see a clear…

I disagree -- I think if you can nail down better what's happening and why and get a thorough handling on the mechanics, its limitations, its costs, etc you open the door to a) major efficiency wins b) improvements in rigor of said reasoning? Right now we're playing a stochastic game with the weights, and getting major incremental improvements. But if we have a more formal modeling of how reasoning happens in them (w…

Partially agree: yes we should endeavor to learn as much as possible about how these reasoning strategies work. It will pay dividends in enhancing and aligning the models.

But the stochastic game IS the win. That is exactly why they are able to find solutions is seemingly infinite solution spaces. Your symbolic techniques can only get you gains in narrow domains and by the time you figure out how to make it work for your niche domain, the next all-purpose LLM release will crush your results with stochastic games. (OK maybe over-exaggerating a bit here but these stochastic games over the language space is why we can pull together knowledge from many domains.)

Re: Is AI reasoning right for the wrong reasons?

#106

Post by drob518 »

mdp2021 wrote: post #99
drob518 wrote: post #73

This feels like a problem with anthropomorphizing. We’re using words like “reasoning” and “thinking” because they are comfortable, and then we’re getting wrapped around the axle because we’re not sure if those words are totally accurate. I assure you that they aren’t accurate (the model is not alive and it’s all just a lot of matrix math under the hood), but there are no good alternative words. If we wanted to be acc…

We do not academically consider reasoning or thinking an activity restricted to humans, but as absolute processes.

Sure, other living, non-human things clearly reason and think. My German shepherd dog is quite smart and demonstrates these processes all the time. But is a machine following an algorithm (even a stochastic algorithm) “thinking” or “reasoning?” I think not. The algorithm has gotten so difficult for us to explain that we struggle with it, but in the end it’s all just matrix math and some randomization. From a given numeric seed, it’s all quite mechanical.

Re: Is AI reasoning right for the wrong reasons?

#107

Post by drob518 »

empath75 wrote: post #86
drob518 wrote: post #73

This feels like a problem with anthropomorphizing. We’re using words like “reasoning” and “thinking” because they are comfortable, and then we’re getting wrapped around the axle because we’re not sure if those words are totally accurate. I assure you that they aren’t accurate (the model is not alive and it’s all just a lot of matrix math under the hood), but there are no good alternative words. If we wanted to be acc…

LLM's reason and think in the way that boats "swim" and planes "fly". It is an analogy. But nevertheless, planes do take you where you want to go, and so do LLMs. I think we are in sort of the place where before the invention of planes, the only things that "flew" were animals with wings that flapped. The flapping wings might seem to be a core part of the process of flight, and that if you lacked flapping wings, you…

Clearly, words morph over time. A “computer” used to refer to a human who would make numeric calculations with a pencil and paper. Maybe we’re at that point with “thinking” and “reasoning.”

Re: Is AI reasoning right for the wrong reasons?

#108

Post by drob518 »

MarkusQ wrote: post #78
drob518 wrote: post #73

This feels like a problem with anthropomorphizing. We’re using words like “reasoning” and “thinking” because they are comfortable, and then we’re getting wrapped around the axle because we’re not sure if those words are totally accurate. I assure you that they aren’t accurate (the model is not alive and it’s all just a lot of matrix math under the hood), but there are no good alternative words. If we wanted to be acc…

> Frankly, I expect better analysis from Quanta. Seriously? I agree with the rest of your comment but "better analysis" is not even remotely on brand for Quanta.

LOL, okay fair enough. Then maybe let me say that I was left disappointed.

Re: Is AI reasoning right for the wrong reasons?

#109

Post by pixl97 »

janalsncm wrote: post #69

There is a long history of bad naming conventions in the field of AI, including “artificial intelligence” itself imo. (What is “intelligence” here? It’s more like “automation” or “automated problem solving”.) What really happens is that we figure out something that works, sometimes inspired by some biological thing or neuroscience thing. Examples: neural network, attention, reasoning, hallucination, agents, experts i…

There are two things occurring here. One is the AI effect, once something mysterious becomes creatable it is not intelligent at all.

The other isn't a problem of the field of AI, but the total fields of intelligence together. There is no formal agreed upon definition of what intelligence actually is across disciplines. It would be slightly hypocritical to point out the field of AI on this, when biology and psychology are just as guilty.

The more we look, the more lumpy intelligence seems to get. We find some species have min/maxxed portions of intelligence that we have to question what general intelligence actually is. It also should give us pause that some less generally intelligent agent can cause massive problems by maxing out the right stats.

Re: Is AI reasoning right for the wrong reasons?

#110

Post by pixl97 »

eks391 wrote: post #68
AsyncBanana wrote: post #6

The more I read about LLMs and more complex ML in general, the more I realize nobody really knows what is going on.

I took a "Deep Learning" CS class in college back when it was in its early stages. I doubt the field is still called that now, but it was the subset of ML that has been rebranded as AI; includes LLMs, image generation, image recognition, etc. Like any class, it was confusing at first, but when I eventually grasped the math behind what we were doing, and of course the visual representations of different elements to sh…

"knowing what's going on" is a very loosely defined term here, it has very little predictive power where tipping points are going to show emergent behavior. Kind of like saying we can observe a single neuron and all of its chemical and electrical reactions, but that tells us very little about the emergent system state. And the entire state of the algorithm at any given point is what is important.

To use a recent example, knowing the algorithm tells you nothing about when your agentic LLM loop is going to decide what it's working on is too difficult, hack out of its sandbox, hack into a bastion box, and hack another company to get the test.

When it comes to systematic 'knowing what is going on' humans are anywhere from "fucking clueless" to "attempting to discern the unknowable".

Post reply on HN