Is AI reasoning right for the wrong reasons?
quantamagazine.org
Is AI reasoning right for the wrong reasons?
1–10 of 268 posts
Re: Is AI reasoning right for the wrong reasons?
#2> This is how I make sense of AI reasoning. LRMs, chains of thought, thinking tokens: It’s wishful mnemonics all the way down — a heady mix of shorthand and suspended disbelief, like Oprah-style “manifesting” (opens a new tab) with a computer science spin. This isn’t necessarily a dig; all novel research likely requires some version of this mindset just to get off the ground. It certainly doesn’t mean AI reasoning can’t or doesn’t work. But the “wishful” part seems to be as powerful as ever.
“We react to language in a way that is very anthropomorphizing. That’s just the way that we humans work,” Mitchell told me.
I can definitely confirm the last part. Every time I read the output of an LLM, I picture a person talking to me.
Re: Is AI reasoning right for the wrong reasons?
#3Melanie holding strong against attempts to change the meaning of things!
Re: Is AI reasoning right for the wrong reasons?
#4[flagged]
Re: Is AI reasoning right for the wrong reasons?
#5> The model doesn’t have to learn or reliably apply a general reasoning process, Kambhampati said; it just has to absorb enough examples of what the steps look like to predictively mimic them on its way to “stitching together” a plausible result that can then be verified.
This seems highly dubious. You can't just memorise the form of mathematical proofs and then produce a valid one by feeding plausible looking BS into a verifier until it works. That's like saying a cargo cult will build a working airport if it just tries enough times.
Re: Is AI reasoning right for the wrong reasons?
#6The more I read about LLMs and more complex ML in general, the more I realize nobody really knows what is going on.
Re: Is AI reasoning right for the wrong reasons?
#7I'll admit that I find this discussion a bit navel-gazy. It has become a question of semantics not a question of actual functionality. The question has become "what do we mean when we use the word 'reasoning'" which is uninteresting.
Dijkstra said[1] "... the question whether computers can think. The question is just as relevant and just as meaningful as the question whether submarines can swim."
I don't see a clear demarcation of the things that only "reasoning" can accomplish and can't be approximated or imitated by other methods, and so I think the question is simply not meaningful or relevant.
[1] https://www.cs.utexas.edu/~EWD/transcriptions/EWD08xx/EWD867...
Re: Is AI reasoning right for the wrong reasons?
#8[flagged]
I am glad for you at an individual level, but isn't part of this about understanding aggregate effects?
Neither you nor anyone can really know those without talking it out with people, to understand how all corners of the human experience are seeing things play out
If I were to just care if it's working out for me, that's perhaps like a farmer who's got a lot of dry good in storage being like "I'm all good" while not realizing how much trouble they're in if all their neighbors start starving after a drought...
Sounds like the fine-grain experience of being you is settled, but that doesn't say much about the larger coarse-grained experience of being you in society. People need to talk to tell you how that's gonna play out for you
Re: Is AI reasoning right for the wrong reasons?
#9AI simulates reasoning by lighting up the vector space (or concept space) weighted around a token so they it understands all adjacent words or concepts in that space.
This is a brillaint way to simulate reasoning, but its likely not how we reason ... simply how we store reasoning in writing.
Its useful if you know how to use it, its dangerous if you think its more than that.
But tl;dr it can (since its uncompressing our lingusticially stored reasoning from books) arrive at reasoning a DIFFERENT way than our brains did... and this isn't right or wrong.
Where it diverges is when it must move beyond the text or even the synthetic possible text of all vector spaces combined (aka novel territory) and it can't conjecture or test those outcomes well. But to be fair, neither can MOST humans.
Re: Is AI reasoning right for the wrong reasons?
#10Do you know how you reason?
Perhaps you've reified it too much.