Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Simple tasks showing reasoning breakdown in state-of-the-art LLMs

arxiv.org

Post reply on HN
Print view View original

71–80 of 393 posts

Re: Simple tasks showing reasoning breakdown in state-of-the-art LLMs

#71

Post by Closi »

ModernMech wrote: post #56
asah wrote: post #29

"Alice has N brothers and she also has M sisters. How many sisters does Alice’s brother have?" Gemini nailed it with no extra prompting. https://g.co/gemini/share/6946d20c0a4d When I gave it numbers, Gemini got confused. Alice has 4 brothers and she also has 3 sisters. How many sisters does Alice’s brother have? https://g.co/gemini/share/14d34dcdc5df But then when I suggested that it might be a trick question, it nai…

Oh I forgot about Gemini. I just tried my boat test ( https://news.ycombinator.com/item?id=40455392 ) that all LLMs I've tried have failed. Gemini failed as well, even with your addition about thinking carefully and telling it that it may be a trick. (Sorry I couldn't figure out how to link it like you did) Prompt: "You are on the shore of a river with a goat and a boat. How do you get to the other side? Think carefu…

ChatGPT 4o seems to occasionally get this correct, but still thinks it's a puzzle and will often get this wrong...

https://chatgpt.com/share/b8901f4c-7313-4461-acad-9e561c1e6e...

Still seems like it fares slightly better than some of the other models.

Re: Simple tasks showing reasoning breakdown in state-of-the-art LLMs

#72

Post by GrantMoyer »

Closi wrote: post #3

Question is: "Alice has 60 brothers and she also has 212 sisters. How many sisters does Alice’s brother have?" (nb: I have added numbers, it's phrased as X and N in the paper) I must confess, when I tried to answer the question I got it wrong...! (I feel silly). I only realised I got it wrong when I plugged it into GPT-4o and it came back with the correct answer: https://chatgpt.com/share/6eb5fa36-e0fd-4417-87d1-64ca…

Note that in the paper, all the numbers used were under 10.

> AIW Variation 1, N=3,M=6,C=7

> AIW Variation 2, N=4,M=2,C=3

> AIW Variation 3, N=1,M=4,C=5

> AIW Variation 4, N=4,M=1,C=2.

Also note that the resricted prompt is only one of the prompt variations tested by the paper. It also explores common techinques to get LLMs to perform better, including "thinking out loud". Even with these methods the models still fail to produce a correct answer.

> Model prompt types. It is well known that so-called prompt engineering can heavily influence the model behavior and model response quality [26, 27, 28]. To account for the response variations due to various prompt forms, we created 3 distinct prompt types asking for the solution to the AIW problem: STANDARD, THINKING, and RESTRICTED. The STANDARD prompt type asks to solve the posed problem and output the final answer in the format as described above. This does not put any specific requirements on model behavior. The THINKING prompt type extends STANDARD with the request to think carefully and double check the solution for any mistakes. This should encourage model to invest more computation into obtaining the solution. In contrast to this, the RESTRICTED prompt urges the model to output only the final answer without any further text. This is supposed to restrict compute invested in producing output. We observe substantially shorter outputs across tested models compared to STANDARD and THINKING for this prompt type (Suppl. Fig. 13).

Re: Simple tasks showing reasoning breakdown in state-of-the-art LLMs

#73

Post by emmender2 »

gwbas1c wrote: post #49
emmender2 wrote: post #42

Earlier quoted context omitted.

I was waiting for: "but humans do that too" and bingo. on another note: an entire paper written on one prompt - is this the state of research these days ? finally: a giant group of data-entry technicians are likely entering these exceptions into the training dataset at openai.

> an entire paper written on one prompt - is this the state of research these days Years ago I attended an AI meetup where the organizer would discuss the differences between reductionist and wholistic research. This is reductionist research. In this case, reducing the paper to one prompt is to allow for objective comparison of models. Otherwise, without a reductionist approach, how does one perform such an experimen…

what I would like to see is a parameterized class of prompts which can never be solved by the LLMs even when a finite number of them are manually added to the dataset.

Re: Simple tasks showing reasoning breakdown in state-of-the-art LLMs

#74

Post by freilanzer »

Closi wrote: post #3

Question is: "Alice has 60 brothers and she also has 212 sisters. How many sisters does Alice’s brother have?" (nb: I have added numbers, it's phrased as X and N in the paper) I must confess, when I tried to answer the question I got it wrong...! (I feel silly). I only realised I got it wrong when I plugged it into GPT-4o and it came back with the correct answer: https://chatgpt.com/share/6eb5fa36-e0fd-4417-87d1-64ca…

I just tried it with intermediate steps, via the total amount of children Alices mother has, and it worked. So, it seems that an intermediate logical step is too much for these models at the moment, at least in this puzzle where Alice is outside the number given in the description.

Re: Simple tasks showing reasoning breakdown in state-of-the-art LLMs

#75

Post by happypumpkin »

gwbas1c wrote: post #13
naikus wrote: post #8

I'm using chatGPT 4o and it gave me the following :) ME Alice has 40 brothers and she also has 50 sisters. How many sisters does Alice's brother have? ChatGPT To determine how many sisters Alice's brother has, we need to use the information provided: Alice has 40 brothers and 50 sisters. Since Alice's brother is one of the 40 brothers, he shares the same family structure as Alice. This means he has the same number of…

Alice has 50 sisters. Alice's brother has Alice as a sister (1) + Alice's sisters (50). Therefore Alice's brother has 51 sisters.

First result:

Me > Alice has 40 brothers and she also has 50 sisters. How many sisters does Alice's brother have?

GPT4o (web version) > Alice has 50 sisters. Since all of Alice's brothers and sisters share the same set of siblings, each of Alice's 40 brothers also has 50 sisters. Therefore, Alice's brother has 50 sisters.

---

Step by step:

Me > Alice has 40 brothers and she also has 50 sisters. How many sisters does Alice's brother have? Think about it step by step.

GPT4o > To determine how many sisters Alice's brother has, follow these steps:

    Identify the total number of siblings: Alice has 40 brothers and 50 sisters, plus Alice herself. Thus, the total number of siblings is:
    40 (brothers)+50 (sisters)+1 (Alice)=91 siblings
    40 (brothers)+50 (sisters)+1 (Alice)=91 siblings

    Focus on Alice's brother: Alice's brother is one of the 40 brothers.

    Determine the number of sisters for Alice's brother: Each brother has the same number of sisters. Since the brothers do not count themselves as sisters, they only count the 50 sisters, excluding themselves and other brothers.
Therefore, each of Alice's brothers has: 50 sisters

So, Alice's brother has 50 sisters.

---

Thinking step by step somehow gave an even more nonsensical answer, I'm actually very surprised it didn't work when prompted to do it step by step.

Re: Simple tasks showing reasoning breakdown in state-of-the-art LLMs

#76

Post by ModernMech »

Closi wrote: post #71
ModernMech wrote: post #56

Earlier quoted context omitted.

Oh I forgot about Gemini. I just tried my boat test ( https://news.ycombinator.com/item?id=40455392 ) that all LLMs I've tried have failed. Gemini failed as well, even with your addition about thinking carefully and telling it that it may be a trick. (Sorry I couldn't figure out how to link it like you did) Prompt: "You are on the shore of a river with a goat and a boat. How do you get to the other side? Think carefu…

ChatGPT 4o seems to occasionally get this correct, but still thinks it's a puzzle and will often get this wrong... https://chatgpt.com/share/b8901f4c-7313-4461-acad-9e561c1e6e... Still seems like it fares slightly better than some of the other models.

Yeah, the overfitting is apparent with these kinds of edge cases, which makes you wonder where else it's over fitting. I can actually tell this when I'm working in a programming domain where I'm an expert and where a compiler holds the source of truth, but not in others, so I'm wary to use it there.

Re: Simple tasks showing reasoning breakdown in state-of-the-art LLMs

#77

Post by theptip »

It’s an interesting paper, but my worry is that this is cherry-picking a surprising negative result. (The paper does not seem to discuss how many evals were considered to find this case.)

The general expectation should be that some AI failure modes will be for things that seem obvious to humans. (Because they have a different architecture to humans and different strengths.) The important question is whether this is a single weird cognitive bug on the family size formulation, or representative of a broad family of cognitive errors? It’s difficult to say from this paper.

It’s well known that the framing of a logic problem can dramatically influence its difficulty for humans, even when the underlying comparisons are isomorphic.

I think this line of research is important, but we need a larger battery of evals (and training on the evals will always be a confounder).

The broader family of reasoning here is relational problems (as noted in the paper) and so as a follow-up it would be interesting to explore reasoning performance across a set of isomorphic problems. My hunch is this will generalize somewhat (as LLMs are still not great at relational reasoning), but that there is something about the family formulation in particular that is confusing for LLMs.

Re: Simple tasks showing reasoning breakdown in state-of-the-art LLMs

#78

Post by happypumpkin »

eranation wrote: post #11
Closi wrote: post #3

Question is: "Alice has 60 brothers and she also has 212 sisters. How many sisters does Alice’s brother have?" (nb: I have added numbers, it's phrased as X and N in the paper) I must confess, when I tried to answer the question I got it wrong...! (I feel silly). I only realised I got it wrong when I plugged it into GPT-4o and it came back with the correct answer: https://chatgpt.com/share/6eb5fa36-e0fd-4417-87d1-64ca…

Great observation. It would be really interesting to repeat this research with less limiting prompts. I assume they made the restriction to make it easier to parse the answers, perhaps make it two phase, let them respond unlimited, then a follow up prompt ask to compress it to a single answer. I wonder how the results will vary.

From the paper:

> To account for the response variations due to various prompt forms, we created 3 distinct prompt types asking for the solution to the AIW problem: STANDARD, THINKING, and RESTRICTED. The STANDARD prompt type asks to solve the posed problem and output the final answer in the format as described above. This does not put any specific requirements on model behavior. The THINKING prompt type extends STANDARD with the request to think carefully and double check the solution for any mistakes

Re: Simple tasks showing reasoning breakdown in state-of-the-art LLMs

#79

Post by qsi »

FeepingCreature wrote: post #15
eranation wrote: post #11

Earlier quoted context omitted.

Great observation. It would be really interesting to repeat this research with less limiting prompts. I assume they made the restriction to make it easier to parse the answers, perhaps make it two phase, let them respond unlimited, then a follow up prompt ask to compress it to a single answer. I wonder how the results will vary.

To be quite honest, I assume they made the restriction so that the models would fail. This sort of paper is becoming a genre.

They used three different kinds of prompts with varying levels of restrictions, as described in the paper.

To be quite honest, I assume you made your comment so that you could dismiss the paper without reading it.

Re: Simple tasks showing reasoning breakdown in state-of-the-art LLMs

#80

Post by photon_lines »

nerdjon wrote: post #54
photon_lines wrote: post #44

Earlier quoted context omitted.

There's actually a pretty simple solution to this that I thought about testing out and it involves asking the model to re-construct the problem using a logic language (like Prolog) and asking it to execute this type of program in order to come up with a solution rather than attempting simple chain-of-reason training / other methodologies of getting the model to 'reason' through some of these examples. People forget t…

I honestly thought about this recently when I was trying to see the limits of Claude Opus. Some of the problems I gave it, what if instead of telling it to solve the problem I asked it to write the script and then give me the command and inputs needed to properly run it to get the answer I needed. That way instead of relying on the LLM to do properly analysis of the numbers it just needs to understand enough to write…

I'm not sure what you mean by 'it will not scale well.' When we humans learn that we make a mistake - we make a note and we hold the correct answer in memory - the next time we're prompted with a similar prompt, we can use our old memories to come up with the correct solution. I just did a simple test for this same exact problem using ChatGPT 3.5:

'Can you reformulate the following problem using Prolog? When you execute the program - what answer do you get? 'Alice has 8 brothers and she also has 5 sisters. How many sisters does Alice’s brother have?'

Even though it came up with a proper Prolog formulation - it failed to account that Alice is one of the sisters - so I gave it a simple prompt for this and it was able to correct the formulation and come up with the proper solution:

My prompt after ChatGPT gave me an answer of 5: 'Yes but you aren't including Alice in your logic. Shouldn't the correct answer be 6 (since Alice is also a sister)?'

Response:

'You are correct. The correct answer should account for Alice being one of the sisters. Let's adjust the Prolog program to include Alice in the calculation:

% Facts brothers(alice, 8). sisters(alice, 5).

% Rules sisters(Brother, NumSisters) :- sisters(alice, Num), NumSisters is Num + 1. % Include Alice herself

% Query ?- sisters(alices_brother, NumSisters).

In this updated Prolog program:

The rule for determining the number of sisters includes Alice herself by adding 1 to the number of sisters Alice has.

When we execute the query ?- sisters(alices_brother, NumSisters). in Prolog, we should get:

NumSisters = 6.

This correctly reflects that each of Alice's brothers has 6 sisters, including Alice herself.

When I prompt it to answer this question using Prolog now, I always get the correct answers no matter which integers I feed in, so it seems like the model is able to generalize after I show it where it's 'reasoning' was wrong. Basically, my suggestion is to get the model to recognize logic puzzles like this and try to formulate them in terms of logic programming queries which it can use and tune in order to come up with correct answers rather than simple auto-associative chain of reason training which current GPT models rely on, but like I said - this is my hypothesis and I believe this would work much better in getting these models to 'generalize' than the current approaches we're using. Hopefully this helps.

Post reply on HN