21–30 of 46 posts
3 videos at https://archive.org/details/Jeopardy_2011-02-14_The_IBM_Chal...
You don’t ask an LLM, but certain LLMs expose internal metrics you can tell how many token candidates where there what was the score which one and which one was selected. So there are objective ways to control hallucinations as well as figuring out how “correct” the answer is to some extent.
is definitely too strong of a claim and directly undercut by what is said near the end of the post: > "Tian et al. found in Just Ask for Calibration that with the right prompting strategy, RLHF’d models verbalize probabilities that are better calibrated than the model’s own conditional probabilities, and that prompting plus temperature scaling can cut expected calibration error by more than half. And Anthropic’s Language Models (Mostly) Know What They Know found encouraging results asking models to estimate the probability that their own proposed answer is true."
My own experience is that stated confidence is a helpful tool and of course you need a rubric and a proper prompt, but this is clearly less work than training a classifier (as advocated by the post) and requires less data.
The statement near the top of the post > "The short version: asking an LLM to generate a score for how confident it is in its own response is, from everything I can tell, completely useless." is definitely too strong of a claim and directly undercut by what is said near the end of the post: > "Tian et al. found in Just Ask for Calibration that with the right prompting strategy, RLHF’d models verbalize probabilities t…
What did work was asking the model to emit citations in the form of line numbers from the transcript. It dramatically dropped hallucination.
I was designing a system that pulls structured data out of an unstructured transcript, and Opus initially suggested a confidence score for each value. When I asked it “do you think the model will generate a meaningful number there?” the answer was basically “ha, you caught me.” What did work was asking the model to emit citations in the form of line numbers from the transcript. It dramatically dropped hallucination.
This also isn't likely to give you good results, models don't have a good sense of their own capabilities.
This is also in the same camp of asking a model why it did something. It'll generate something plausible and likely totally incorrect.
The statement near the top of the post > "The short version: asking an LLM to generate a score for how confident it is in its own response is, from everything I can tell, completely useless." is definitely too strong of a claim and directly undercut by what is said near the end of the post: > "Tian et al. found in Just Ask for Calibration that with the right prompting strategy, RLHF’d models verbalize probabilities t…
I was designing a system that pulls structured data out of an unstructured transcript, and Opus initially suggested a confidence score for each value. When I asked it “do you think the model will generate a meaningful number there?” the answer was basically “ha, you caught me.” What did work was asking the model to emit citations in the form of line numbers from the transcript. It dramatically dropped hallucination.
> I asked it “do you think the model will generate a meaningful number there?” This also isn't likely to give you good results, models don't have a good sense of their own capabilities. This is also in the same camp of asking a model why it did something. It'll generate something plausible and likely totally incorrect.
It actually had some memorized knowledge about LLMs being bad at confidence intervals, then it did a cursory web search. But it didn't need to do any of that; I already knew they would be bad numbers.