Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs get lost in multi-turn conversation

arxiv.org

Post reply on HN
Print view View original

41–50 of 272 posts

Re: LLMs get lost in multi-turn conversation

#41

Post by [deleted] »

CompoundEyes wrote: post #27
Benjammer wrote: post #2

It's nice to see a paper that confirms what anyone who has practiced using LLM tools already knows very well, heuristically. Keeping your context clean matters, "conversations" are only a construct of product interfaces, they hurt the quality of responses from the LLM itself, and once your context is "poisoned" it will not recover, you need to start fresh with a new chat.

Agreed poisoned is a good term. I’d like to see “version control” for conversations via the API and UI that lets you rollback to a previous place or clone from that spot into a new conversation. Even a typo or having to clarify a previous message skews the probabilities of future responses due to the accident.

[deleted]

Re: LLMs get lost in multi-turn conversation

#42

Post by djmips »

Benjammer wrote: post #2

It's nice to see a paper that confirms what anyone who has practiced using LLM tools already knows very well, heuristically. Keeping your context clean matters, "conversations" are only a construct of product interfaces, they hurt the quality of responses from the LLM itself, and once your context is "poisoned" it will not recover, you need to start fresh with a new chat.

Happens with people too if you think about it.

Re: LLMs get lost in multi-turn conversation

#43

Post by zacksiri »

cadamsdotcom wrote: post #34
zacksiri wrote: post #11

I've been working on solving this with quite a bit of success, I'll be sharing more on this soon. It involves having 2 systems 1st system is the LLM itself and another system which acts like a 'curator' of thoughts you could say. It dynamically swaps in / out portions of the context. This system is also not based on explicit definitions it relies on LLMs 'filling the gaps'. The system helps the llm break down problem…

Sounds like an exciting idea. May I suggest - put what you have out there in the world, even if it’s barely more than a couple of prompts. If people see it and improve on it, and it’s a good idea, it’ll get picked up & worked on by others - might even take on a life of its own!

Have a look here, it's an early preview

https://x.com/zacksiri/status/1922500206127349958

You can see it's going from introduction, asking me for my name, and then able to answer question about some topic. There is also another example in the thread you can see.

Behind the scenes, the system prompt is being modified dynamically based on the user's request.

All the information about movies is also being loaded into context dynamically. I'm also working on some technique to unload stuff from context when the subject matter of a given thread has changed dramatically. Imagine having a long thread of conversation with your friend, and along the way you 'context switch' multiple times as time progresses, you probably don't even remember what you said to your friend 4 years ago.

There is a concept of 'main thread' and 'sub threads' involved as well that I'm exploring.

I will be releasing the code base in the coming months. I need to take this demo further than just a few prompt replies.

Re: LLMs get lost in multi-turn conversation

#44

Post by voidspark »

Sharlin wrote: post #7

Seems like this is an aspect of their well-known overconfidence and the inability to self-reflect and recognize they have to ask for more details because their priors are too low. If you look at the output of reasoning models, it’s clear that the idea of asking for clarification very rarely occurs to them – when they’re confused, it’s just endless speculation of what the user might have meant. This, of course, has ce…

> inability to self-reflect and recognize they have to ask for more details because their priors are too low.

Gemini 2.5 Pro and ChatGPT-o3 have often asked me to provide additional details before doing a requested task. Gemini sometimes comes up with multiple options and requests my input before doing the task.

Re: LLMs get lost in multi-turn conversation

#45

Post by Helmut10001 »

olalonde wrote: post #32
Helmut10001 wrote: post #14

Earlier quoted context omitted.

My experiences somewhat confirm these observations, but I also had one that was different. Two weeks of debugging IPSEC issues with Gemini. Initially, I imported all the IPSEC documentation from OPNsense and pfSense into Gemini and informed it of the general context in which I was operating (in reference to 'keeping your context clean'). Then I added my initial settings for both sides (sensitive information redacted!…

Recently, Gemini helped me fix a bug in a PPP driver (Zephyr OS) without prior knowledge of PPP or even driver development really. I would copy-paste logs of raw PPP frames in HEX and it would just decode everything and explain the meaning of each bytes. In about an hour, I knew enough about PPP to fix the bug and submit a patch. https://g.co/gemini/share/7edf8fa373fe

Yes, it fells like setting the `-h` flag for logs (human readable).

Re: LLMs get lost in multi-turn conversation

#46

Post by bigcat12345678 »

sitkack wrote: post #39
Terr_ wrote: post #22

Earlier quoted context omitted.

It means if you want something resembling a self-introspective theory of mind, you need to arrange the overall document to cohere to documents where such things are/appear-to-be happening. This leads us to new questions: How can we characterize and identify real-world documents which fit? How can we determine what features may be significant, and which of those can be easily transplanted to our use-case?

You are just doubling down on protecting your argument. I operate LLMs in many conversational modes where it does ask clarifying questions, probing questions, baseline determining questions. It takes at most one sentence in the prompt to get them to act this way.

> It takes at most one sentence in the prompt to get them to act this way.

What is this one sentence you are using?

I am struggling to elicite clarification behavior form llms

Re: LLMs get lost in multi-turn conversation

#47

Post by bobsyourbuncle »

Sharlin wrote: post #7

Seems like this is an aspect of their well-known overconfidence and the inability to self-reflect and recognize they have to ask for more details because their priors are too low. If you look at the output of reasoning models, it’s clear that the idea of asking for clarification very rarely occurs to them – when they’re confused, it’s just endless speculation of what the user might have meant. This, of course, has ce…

Isn’t this relatively trivial to correct? Just like chain of thought reasoning replaces end tokens with “hmm” to continue the thought can’t users just replace the llm tokens whenever it starts saying “maybe they are referring to” with something like. “Let me ask a clarifying question before I proceed.”

Re: LLMs get lost in multi-turn conversation

#48

Post by QuadmasterXLII »

unshavedyak wrote: post #16
Benjammer wrote: post #2

It's nice to see a paper that confirms what anyone who has practiced using LLM tools already knows very well, heuristically. Keeping your context clean matters, "conversations" are only a construct of product interfaces, they hurt the quality of responses from the LLM itself, and once your context is "poisoned" it will not recover, you need to start fresh with a new chat.

Has any interface implemented a .. history cleaning mechanism? Ie with every chat message focus on cleaning up dead ends in the conversation or irrelevant details. Like summation but organic for the topic at hand? Most history would remain, it wouldn’t try to summarize exactly, just prune and organize the history relative to the conversation path?

the problem is that it needs to read the log to prune the log, and so if there is garbage in the log, which needs to be pruned to keep from poisoning the main chat, then the garbage will poison the pruning model, and it will do a bad job pruning.

Re: LLMs get lost in multi-turn conversation

#49

Post by airylizard »

Why I came up with TSCE(Two-Step Contextual Enrichment).

+30pp uplift when using GPT-35-turbo on a mix of 300 tasks.

Free open framework, check the repo try it yourself

https://github.com/AutomationOptimization/tsce_demo

I tested this another 300 times with gpt-4.1 to remove those obtrusive "em-dashes" everyone hates. Tested a single-pass baseline vs TSCE, same exact instructions and prompt "Remove the em-dashes from my linkedin post. . .".

Out of the 300 tests, baseline failed to remove the em-dashes 149/300 times. TSCE failed to remove the em-dashes 18/300 times.

It works, all the data as well as the entire script used for testing is in the repo.

Re: LLMs get lost in multi-turn conversation

#50

Post by rrr_oh_man »

Benjammer wrote: post #21
unshavedyak wrote: post #16

Earlier quoted context omitted.

Has any interface implemented a .. history cleaning mechanism? Ie with every chat message focus on cleaning up dead ends in the conversation or irrelevant details. Like summation but organic for the topic at hand? Most history would remain, it wouldn’t try to summarize exactly, just prune and organize the history relative to the conversation path?

I mean, you could build this, but it would just be a feature on top of a product abstraction of a "conversation". Each time you press enter, you are spinning up a new instance of the LLM and passing in the entire previous chat text plus your new message, and asking it to predict the next tokens. It does this iteratively until the model produces a token, and then it returns the text to you and the PRODUCT parses it ba…

Or delete / edit messages in AI Studio or Open Router.
Post reply on HN