What's the best explanation for LLMs being almost right when citing sources?

For example, I asked GPT-4 to identify the source for a quote from a short story by Edgar Allan Poe. It correctly identified the author but not the title. What's the best available explanation for such behavior by LLMs?

Perhaps... but "prompt engineering" right now seems like throwing paint at a wall until the black box evaluates to the relative "truth" you were looking for. It's like a stochastic wrench you turn until it serves your intensive porpoises.

Tokens? The way I see these things operating (in my head) is as a hyper-dimensional merge sort which lose there context/bounded-domain during evaluation, leading to something less than the sum of its parts because the weights between tokens correlate linguistic/phonetic relationships--which lose their causal-relationship to the real world.

I had a prompt saved which would give full sources per sentence of response. It was useful for one purpose then became annoying and time consuming. I was diagnosing hallucinations and training data issues.

Maybe crafting something to give a full APA or MLA citation and works cited page per response could help.

https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00563...

The prompt isn't relevant to this question though. The quality of output can be improved with better input but in this case, I am curious about the underlying mechanics in the model that leads to such behavior.

LLMs are auto-regressive predictors -- so they take the text given to them (i.e., the prompt) and generate a probability estimate for what the next token should be.

Suppose you gave it a quote -- "Once upon a midnight dreary, while I pondered, " and ask it to keep writing, it will generate a probability distribution across various tokens it has been trained on.

I'll use words here, rather than tokens, to make the point... Hypothetically, for the quote above, the LLM might estimate the probability of the next word being...

"Weak" = 0.80

"Tired" = 0.10

"Slothful" = 0.05

... and so on.

Now, if you are using a temperature of 0.0, the LLM will pick the highest probability word/token. It's possible you had a non-zero temperature setting and the LLM "knew" the right answer but randomly picked the wrong one... Temperature basically randomizes the token choice to make it more diverse/creative/better.

Alternatively, based on the prompt (i.e., collection of text you put in), it estimated the probability of the wrong answer to be higher. In your case, the LLM likely had a high probability for "Edgar Allan Poe" but maybe a lower probability for the specific works/titles, and hence chose incorrectly.

P.S.

If you are using the OpenAI playground, you can actually get the probability estimates, if you want to investigate further!