Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Simple tasks showing reasoning breakdown in state-of-the-art LLMs

arxiv.org

Post reply on HN
Print view View original

41–50 of 393 posts

Re: Simple tasks showing reasoning breakdown in state-of-the-art LLMs

#41

Post by nerdjon »

mjburgess wrote: post #33
nerdjon wrote: post #18

For anyone considering reading the paper and like me don't normally read papers like this, open the PDF and think you don't have time to read it due to its length. The main part of the paper is the first 10 pages and a fairly quick read. On to the topic here. This is an interesting example that they are using. It is fairly simplistic to understand as a human (even if we may be inclined to quickly jump to the wrong co…

In many ways, this is very obvious and routine to people who use these systems with a critical understanding of how they work. It's dispiriting how we require a arxiv.org pdf in order to point out the absence of the emperor's clothing. There are few people who use LLMs who could not, right now, demonstrate this point to themselves if they so wish. Such is the strength of corporate tech propaganda that a whole mass of…

> In many ways, this is very obvious and routine to people who use these systems with a critical understanding of how they work.

The last part of that is the problem and why a paper like this is critical.

These systems are being pushed onto people who don't understand how they work. CEO's and other business leaders are being pushed to use AI. Average users are being shown it in Google search results. Etc etc.

People are being told it can do far more than it really is.

Re: Simple tasks showing reasoning breakdown in state-of-the-art LLMs

#42

Post by emmender2 »

gwbas1c wrote: post #9

> This breakdown can be considered to be dramatic ... because models tend to express strong overconfidence in reporting their wrong solutions as correct, while often providing confabulations to additionally explain the provided final answer, mimicking reasoning-like tone but containing nonsensical arguments as backup for the equally nonsensical, wrong final answers. People do that too! Magical thinking is one example…

I was waiting for: "but humans do that too" and bingo.

on another note: an entire paper written on one prompt - is this the state of research these days ?

finally: a giant group of data-entry technicians are likely entering these exceptions into the training dataset at openai.

Re: Simple tasks showing reasoning breakdown in state-of-the-art LLMs

#44

Post by photon_lines »

nerdjon wrote: post #18

For anyone considering reading the paper and like me don't normally read papers like this, open the PDF and think you don't have time to read it due to its length. The main part of the paper is the first 10 pages and a fairly quick read. On to the topic here. This is an interesting example that they are using. It is fairly simplistic to understand as a human (even if we may be inclined to quickly jump to the wrong co…

There's actually a pretty simple solution to this that I thought about testing out and it involves asking the model to re-construct the problem using a logic language (like Prolog) and asking it to execute this type of program in order to come up with a solution rather than attempting simple chain-of-reason training / other methodologies of getting the model to 'reason' through some of these examples. People forget that humans don't come up with their logical models out of the blue - it takes years of elementary school in order for us to understand the world and solve problems in it. The logic programming approach I'd say is really promising but you would need to feed the LLM a LOT of examples in order for it to work, and currently I'm not even sure that we have enough training data in order to implement something like this.

Re: Simple tasks showing reasoning breakdown in state-of-the-art LLMs

#46

Post by nerdjon »

colechristensen wrote: post #40
nerdjon wrote: post #18

For anyone considering reading the paper and like me don't normally read papers like this, open the PDF and think you don't have time to read it due to its length. The main part of the paper is the first 10 pages and a fairly quick read. On to the topic here. This is an interesting example that they are using. It is fairly simplistic to understand as a human (even if we may be inclined to quickly jump to the wrong co…

I don't think anybody who actually uses LLMs thinks they are general purpose AI. Like you ask it to do one thing it's amazing, but then you try to modify or do something with extra steps, or just anything with any complexity to it and it falls over.

> I don't think anybody who actually uses LLMs thinks they are general purpose AI.

I would like to believe that but I have had too many conversations with people who basically think it already is. Including in one situation of a fellow engineer.

It feels like more and more "we" are in a bubble of actually having some knowledge of how this works, what the actual limitations are, and what it just is not. While there is in fact criticism of it out there, particularly around AI "art". It doesn't seem to be focused on the area we are talking about.

Re: Simple tasks showing reasoning breakdown in state-of-the-art LLMs

#47

Post by mjburgess »

nerdjon wrote: post #41
mjburgess wrote: post #33

Earlier quoted context omitted.

In many ways, this is very obvious and routine to people who use these systems with a critical understanding of how they work. It's dispiriting how we require a arxiv.org pdf in order to point out the absence of the emperor's clothing. There are few people who use LLMs who could not, right now, demonstrate this point to themselves if they so wish. Such is the strength of corporate tech propaganda that a whole mass of…

> In many ways, this is very obvious and routine to people who use these systems with a critical understanding of how they work. The last part of that is the problem and why a paper like this is critical. These systems are being pushed onto people who don't understand how they work. CEO's and other business leaders are being pushed to use AI. Average users are being shown it in Google search results. Etc etc. People…

Sure, but even these people... the failures are so common, and often very obvious. Consider a CEO who puts a press briefing in and asks some questions about it, it's not uncommon for those answers to be obviously wrong on any sort of critical reflection.

We arent dealing with a technology that is 99.9% right in our most common use cases, so that we need to engineer some incredibly complex problem to expose the flaw. Rather, in most cases there is some obvious flaw. It's a system that requires typically significant "prompt engineering" to provide the reasoning the system otherwise lacks.

I guess that offers an explanation: people aren't aware that via their own prompt engineering they are repairing the deficiencies of the process by manipulating its inputs to include the structured reasoning it lacks. So there's a sort of hot-reading effect at work.

Re: Simple tasks showing reasoning breakdown in state-of-the-art LLMs

#48

Post by Closi »

nerdjon wrote: post #37
Closi wrote: post #3

Question is: "Alice has 60 brothers and she also has 212 sisters. How many sisters does Alice’s brother have?" (nb: I have added numbers, it's phrased as X and N in the paper) I must confess, when I tried to answer the question I got it wrong...! (I feel silly). I only realised I got it wrong when I plugged it into GPT-4o and it came back with the correct answer: https://chatgpt.com/share/6eb5fa36-e0fd-4417-87d1-64ca…

Even "thinking out loud" we have seen these systems fall flat on their face and give very wrong answers. Thinking out loud also only gets you so far, if the expectation is a certain type of response it can't always "think out loud". In reality that just proves it isn't really reasoning here and is more likely just self referencing. That being said, I tried this personally allowing it to think out loud and it told me…

> Thinking out loud also only gets you so far, if the expectation is a certain type of response it can't always "think out loud". In reality that just proves it isn't really reasoning here and is more likely just self referencing.

Try to calculate it without writing anything down, or thinking any numbers or words in your head.

You can't draw a 1:1 analogue between an AI and the human experience, but remember that we have an internal stream of consciousness. Maybe the outputs of the LLM are more similar to the stream of consciousness in our heads rather than the words we say? After all, Humans also do lots of self referencing.

> That being said, I tried this personally allowing it to think out loud and it told me she has 212 sisters. Using your exact prompt.

Fair enough, but worst case it can often solve it correctly with the correct reasoning. GPT3.5 can't solve it correctly with correct reasoning, so we are at least appearing to be on a path where AI's can start to solve this question, albeit potentially not fully reliably.

Re: Simple tasks showing reasoning breakdown in state-of-the-art LLMs

#49

Post by gwbas1c »

emmender2 wrote: post #42
gwbas1c wrote: post #9

> This breakdown can be considered to be dramatic ... because models tend to express strong overconfidence in reporting their wrong solutions as correct, while often providing confabulations to additionally explain the provided final answer, mimicking reasoning-like tone but containing nonsensical arguments as backup for the equally nonsensical, wrong final answers. People do that too! Magical thinking is one example…

I was waiting for: "but humans do that too" and bingo. on another note: an entire paper written on one prompt - is this the state of research these days ? finally: a giant group of data-entry technicians are likely entering these exceptions into the training dataset at openai.

> an entire paper written on one prompt - is this the state of research these days

Years ago I attended an AI meetup where the organizer would discuss the differences between reductionist and wholistic research.

This is reductionist research.

In this case, reducing the paper to one prompt is to allow for objective comparison of models.

Otherwise, without a reductionist approach, how does one perform such an experiment and ensure that they are comparing apples to apples?

Re: Simple tasks showing reasoning breakdown in state-of-the-art LLMs

#50

Post by James_K »

colechristensen wrote: post #40
nerdjon wrote: post #18

For anyone considering reading the paper and like me don't normally read papers like this, open the PDF and think you don't have time to read it due to its length. The main part of the paper is the first 10 pages and a fairly quick read. On to the topic here. This is an interesting example that they are using. It is fairly simplistic to understand as a human (even if we may be inclined to quickly jump to the wrong co…

I don't think anybody who actually uses LLMs thinks they are general purpose AI. Like you ask it to do one thing it's amazing, but then you try to modify or do something with extra steps, or just anything with any complexity to it and it falls over.

> I don't think anybody who actually uses LLMs thinks they are general purpose AI.

They are being sold as such. Most people don't know anything about the topic and will buy that marketing. The entire concept of these models is that you can put a whole bunch of data in and eventually some kind of magic will happen and you get AGI out. They would not see the kind of investment that they do if all that was being promised was "really good predictive text". In fact some philosophers argue that sentience is just really good predictive text to try and make the point that these models are AGI.

Post reply on HN