The raw runs are useful, but plausibility alone seems like a weak way to distinguish memorization from hallucination. I’d be curious to see a control set using random name pairs and the same number of trials. Then you could compare how often rare names, exact phrases, dates, or other specific details recur across fresh sessions. If “Dario and Amanda” produces stable, uncommon fragments while the controls only produce…
The content is hallucinated. LLM training doesn’t lend itself to easy training corpus document retrieval in that way. It’s not impossible to get segments nearly verbatim on a small model with temperature at 0 and a unique starting prompt for continuation, but that’d be more of a one-off on a carefully crafted prompt. It has been done, but in the “researchers show it’s possible to get something verbatim”, not “retriev…
It seems this narrative is filled with anxiety, fear and stress.