Reducing Doom Loops with Final Token Preference Optimization
1–10 of 11 posts
Re: Reducing Doom Loops with Final Token Preference Optimization
#2It would be great if this could be combined with quantization-aware finetuning. In my experience, Qwen3.6-27B has much fewer repetitions at Q6 quantization level as compared to e.g. Q4, but that leaves little space for context on my 24GB RTX 3090.
Re: Reducing Doom Loops with Final Token Preference Optimization
#3this is pretty cool. i think part of the root cause is current rlhf post training design around confidence and optics rather than cooperative transparent honesty. though its kinda an expensive hypothesis to dig into as a private individual
Re: Reducing Doom Loops with Final Token Preference Optimization
#4It would be great if this could be combined with quantization-aware finetuning. In my experience, Qwen3.6-27B has much fewer repetitions at Q6 quantization level as compared to e.g. Q4, but that leaves little space for context on my 24GB RTX 3090.
They are orthogonal; preference optimization like RLHF can be done on the base model which can later be quantized, or it could be done on a new LoRA that is then converted to QLoRA.
Re: Reducing Doom Loops with Final Token Preference Optimization
#5this is pretty cool. i think part of the root cause is current rlhf post training design around confidence and optics rather than cooperative transparent honesty. though its kinda an expensive hypothesis to dig into as a private individual
Most of the models where people are concerned about don't do this when unquantized, so I doubt it's much about the metapolitics imposed in reinforcement training.
Re: Reducing Doom Loops with Final Token Preference Optimization
#6I'm curious if this approach can be generalized beyond "doom loops".
For instance, another way of thinking about a "doom loop" is wasted tokens, which happens all the time with larger models that are inefficient at test time. Can "bad-ish" tokens be identified and penalized?
Maybe this is already SOTA but would love to learn more!
Re: Reducing Doom Loops with Final Token Preference Optimization
#7Interesting approach. From the practitioner side, the doom loop pattern I hit most with coding agents is context pollution: after two or three failed fix attempts, the failed approaches stay in context and the model keeps circling them. Clearing the session and rewriting the prompt with the exact symptom almost always beats a fourth correction. Curious whether penalizing at the token preference level actually reduces that, or whether it mostly shifts where the loop shows up, since the root cause often seems to be the conversation history rather than the model's token-level tendencies.
Re: Reducing Doom Loops with Final Token Preference Optimization
#8> On an early checkpoint of LFM2.5-2.6B, 10.2% of completions on hard math and coding prompts produced repetitive loops. After Antidoom training, that rate fell to 1.4%, with eval scores improving across the board as a direct result of reduced looping.
That's a 9% reduction, and it doesn't really discuss applicability over longer contexts, different language/language familes/programming, or context/rope length aspects.
Several families - esp Llama and gemma - seem to be easily doom-inclined during expansions (I realize this is a pathological misuse of the tooling, but it's what people do): prompt: "add comments to this file", +1000t file read, [+3k reasoning], +800t edits, +1100t file re-read promoted by tool, prompt:"pep8 format it", [+2kt reasoning], +700t edits, +950t file re-read.
The unmodified/repeating sequences in those file reads, the differences between them, is a honeypot for attention.
I'd non-scientifically concluded for myself that this was tied to an anti-pattern for rope. In the first portion of the context inference needs attention to fall in certain places to produce the correct suggestions for the first round of edits, and somehow largely refocus attention to the subsequent version of the file in the context to correctly apply the subsequent edits, not getting distracted by the tantalizing pattern of repeated sequences and disrupted sequences
Using Gemma4 e2b I can repro this in LM-Studio, and fix it by hand-editing the context so that there's only the latest version of the file being worked on up at the start of the context, or replacing the original reads with something like:
" [tool name='read-file' path=...']... elided, see file.v1.py if you genuinely need to revisit the original state...[/tool] "
The handful of times I've tried this over the last 3 years it hasn't cared to discover file.v1.py didnt exist, limited samples = limited confidence.
Re: Reducing Doom Loops with Final Token Preference Optimization
#9Interesting approach. From the practitioner side, the doom loop pattern I hit most with coding agents is context pollution: after two or three failed fix attempts, the failed approaches stay in context and the model keeps circling them. Clearing the session and rewriting the prompt with the exact symptom almost always beats a fourth correction. Curious whether penalizing at the token preference level actually reduces…
I've got a shim that will ask the model to write a summary of what didn't work, then rolls it rolls back the context to before those attempts with only the summary as advice. Some care has to be taken with cache management that there is an available checkpoint back there.
Appears to help, but I suspect it would be much better with first class support and reinforcement learning to make good use of this kind of advice.
Still certainly needs some tuning because I've noticed the model taking the advice as the word of god and avoiding trying anything remotely like the things that didn't work.
Re: Reducing Doom Loops with Final Token Preference Optimization
#10Somewhere around the initial ChatGPT release, I was experimenting with having the model draw various types of ASCII art. I believe the first time I ran into a doom loop might have been when I asked it to draw a giraffe; it started with a head, then a line of | | for the neck, and then it just repeated that, ad infinitum. I am not really sure what the cause was, but clearly it wanted to make a long-ish neck, and ran out of a way to determine how many lines it had already produced, so just kept going.