Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs get lost in multi-turn conversation

arxiv.org

Post reply on HN
Print view View original

71–80 of 272 posts

Re: LLMs get lost in multi-turn conversation

#71

Post by roywiggins »

petesergeant wrote: post #60
Sharlin wrote: post #7

Seems like this is an aspect of their well-known overconfidence and the inability to self-reflect and recognize they have to ask for more details because their priors are too low. If you look at the output of reasoning models, it’s clear that the idea of asking for clarification very rarely occurs to them – when they’re confused, it’s just endless speculation of what the user might have meant. This, of course, has ce…

> and the inability to self-reflect and recognize they have to ask for more details They're great at both tasks, you just have to ask them to do it.

You can certainly convince them to ask for details, but I'm not sure whether that makes them any good at knowing when exactly to ask vs just asking some percentage of the time regardless.

That is, does it actually know when it doesn't know, or are you just making it less confident overall, so it asks questions with no actual insight? Convincing a model to roleplay as someone who doesn't know things vs teaching a model to have insight into when it does and doesn't need clarification seems like a tough one.

Re: LLMs get lost in multi-turn conversation

#72

Post by gdudeman »

CompoundEyes wrote: post #27
Benjammer wrote: post #2

It's nice to see a paper that confirms what anyone who has practiced using LLM tools already knows very well, heuristically. Keeping your context clean matters, "conversations" are only a construct of product interfaces, they hurt the quality of responses from the LLM itself, and once your context is "poisoned" it will not recover, you need to start fresh with a new chat.

Agreed poisoned is a good term. I’d like to see “version control” for conversations via the API and UI that lets you rollback to a previous place or clone from that spot into a new conversation. Even a typo or having to clarify a previous message skews the probabilities of future responses due to the accident.

This exists in Claude. Edit any previous message and it will fork the conversation.

Re: LLMs get lost in multi-turn conversation

#73

Post by gdudeman »

mh- wrote: post #28
CompoundEyes wrote: post #27

Earlier quoted context omitted.

Agreed poisoned is a good term. I’d like to see “version control” for conversations via the API and UI that lets you rollback to a previous place or clone from that spot into a new conversation. Even a typo or having to clarify a previous message skews the probabilities of future responses due to the accident.

"Forking" or "branching" (probably better received outside of SWEs) a conversation really ought to be a first class feature of ChatGPT et Al.

It is!

It exists in Claude as a true branch - you can see the old threads - and in ChatGPT as without the history.

Edit a previous reply and hit “go” to see it in action.

Re: LLMs get lost in multi-turn conversation

#74

Post by jcims »

Sharlin wrote: post #20
Terr_ wrote: post #17

Earlier quoted context omitted.

> inability to self-reflect IMO the One Weird Trick for LLMs is recognizing that there's no real entity, and that users are being tricked into a suspended-disbelief story. In most cases cases you're contributing text-lines for a User-character in a movie-script document, and the LLM algorithm is periodically triggered to autocomplete incomplete lines for a Chatbot character. You can have an interview with a vampire D…

This is a tired semantic argument that does not bring any insight into the discussion. A token-predictor could still be trained to predict the tokens “I’m not sure what you mean because of points x, y, and z; could you elaborate?”

I agree that it's a tired argument, but there appears to be two separate things being discussed in this little corner of HN. Clarity in the problem it's being asked to solve, and confidence that the answer it has is correct.

I can trivially get any of the foundational models to ask me clarifying questions. I've never had one respond with 'I don't know'.

Re: LLMs get lost in multi-turn conversation

#75

Post by bytepoet »

Sharlin wrote: post #7

Seems like this is an aspect of their well-known overconfidence and the inability to self-reflect and recognize they have to ask for more details because their priors are too low. If you look at the output of reasoning models, it’s clear that the idea of asking for clarification very rarely occurs to them – when they’re confused, it’s just endless speculation of what the user might have meant. This, of course, has ce…

The inability of LLMs of ask for clarification was exactly the flaw we encountered when testing them on open-ended problems, stated somewhat ambiguously. This was in the context of paradoxical situations, tested on DeepSeek-R1 and Claude-3.7-Sonnet. Blog post about our experiments: https://pankajpansari.github.io/posts/paradoxes/

Re: LLMs get lost in multi-turn conversation

#76

Post by jsemrau »

That's no surprise. When I was working on game theory and agent reasoning I reached the same conclusion a year ago.

My conclusion was that context needs to be managed well for the LLMs to manage accuracy in replies. Also, it helps to have a planning process ("graph reasoning") before task execution because it guardrails the models thought process.

This also introduces a discussion on general use vs workflow agent implementations as in the former it is much more difficult to generalize all components in structuring effective ReAct patterns.

Re: LLMs get lost in multi-turn conversation

#77

Post by somenameforme »

morsecodist wrote: post #9
Benjammer wrote: post #2

It's nice to see a paper that confirms what anyone who has practiced using LLM tools already knows very well, heuristically. Keeping your context clean matters, "conversations" are only a construct of product interfaces, they hurt the quality of responses from the LLM itself, and once your context is "poisoned" it will not recover, you need to start fresh with a new chat.

This matches my experience exactly. "poisoned" is a great way to put it. I find once something has gone wrong all subsequent responses are bad. This is why I am iffy on ChatGPT's memory features. I don't notice it causing any huge problems but I don't love how it pollutes my context in ways I don't fully understand.

It's interesting how much the nature of LLMs fundamentally being self recursive next token predictors aligns with the Chinese Room experiment. [1] In such experiment it also makes perfect sense that a single wrong response would cascade into a series of subsequent ever more drifting errors. I think it all emphasizes the relevance of the otherwise unqualifiable concept of 'understanding.'

In many ways this issue could make the Chinese Room thought experiment even more compelling. Because it's a very practical and inescapable issue.

[1] - https://en.wikipedia.org/wiki/Chinese_room

Re: LLMs get lost in multi-turn conversation

#78

Post by 9dev »

CobrastanJorji wrote: post #62
Benjammer wrote: post #2

It's nice to see a paper that confirms what anyone who has practiced using LLM tools already knows very well, heuristically. Keeping your context clean matters, "conversations" are only a construct of product interfaces, they hurt the quality of responses from the LLM itself, and once your context is "poisoned" it will not recover, you need to start fresh with a new chat.

An interesting little example of this problem is initial prompting, which is effectively just a permanent, hidden context that can't be cleared. On Twitter right now, the "Grok" bot has recently begun frequently mentioning "White Genocide," which is, y'know, odd. This is almost certainly because someone recently adjusted its prompt to tell it what its views on white genocide are meant to be, which for a perfect chatb…

Well, telling an AI chatbot to insist on discussing a white genocide seems like a perfectly Elon thing to do!

Re: LLMs get lost in multi-turn conversation

#79

Post by silisili »

coderatlarge wrote: post #24

i’ve see deepseek-coder local get into an infinite loop generating the same line over and over. which i assume without evidence is some sort of feedback from the generated line back into the generation process. so kind of getting lost in thought and going off topic from the simple .h api that my prompt asked for.

Yes! Deepseek does this to me all the time.

I had 20 something files I wanted it to check and change something. The first 5 or so it did, then the sixth it rightly said everything is correct moving on. It said that for the rest of the 20, the same text over and over.

I checked, and file 6 was the only correct one. It like, learned to just repeat itself after that and did nothing.

Re: LLMs get lost in multi-turn conversation

#80

Post by layer8 »

mh- wrote: post #28
CompoundEyes wrote: post #27

Earlier quoted context omitted.

Agreed poisoned is a good term. I’d like to see “version control” for conversations via the API and UI that lets you rollback to a previous place or clone from that spot into a new conversation. Even a typo or having to clarify a previous message skews the probabilities of future responses due to the accident.

"Forking" or "branching" (probably better received outside of SWEs) a conversation really ought to be a first class feature of ChatGPT et Al.

This has been in ChatGPT from pretty early on? Just edit any prompt, it creates a new branch, and you can switch back and forth.
Post reply on HN