111–120 of 143 posts
Earlier quoted context omitted.
An LLM is an autocorrect in as much as humans are replicators. Something seriously gets lost in this "explanation".
What are humans, fundamentally, then ?
But I think we still do not know.
Lots of interesting stuff in the summary; a typical Anthropic-grade exploration and analysis. Thanks you guys! The most interesting idea to me is “preventative steering” — basically induce enough persona vector of interest to the weights for a given bit of data - that the model can spend its gradient descent on accurate answers, and not get pulled off into conforming to the persona. This apparently works, and keeps t…
Earlier quoted context omitted.
Furthermore, it is very rare to have the following kind of text present in the training data: "What is the answer to X?" - "I don't know, I am not sure." In this situation very often there won't be _any_ answer, plenty of difficult questions go unanswered on the internet. Yet the model probably does not interpret this scenario as such
Has anybody tried what seems obvious? Have a series of pretraining sessions with training data where specific information is not present and training questions/answers of "I don't know" for that data is also trained on. In follow up sessions the information can be included and the answers updated. Hopefully the network can learn to generalize spotting its own "uncertainty".
> Other personality changes are subtler but still unsettling, like when models start sucking up to users or making up facts. My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement. The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function…
LLM can be trained to produce "I don't know" when confidence in other answers is weak (e.g. weak or mixed signals). Persona vector can also nudge it into that direction.
> Other personality changes are subtler but still unsettling, like when models start sucking up to users or making up facts. My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement. The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function…
Hallucinations are beginning to appear as a cognitive bias or cognitive deficiency in it's intelligence which is more of an architectural problem rather than a statistics oriented one.
> Other personality changes are subtler but still unsettling, like when models start sucking up to users or making up facts. My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement. The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function…
Sucking up does appear to be a personality trait. Hallucinations are not a completely known or well understood yet. We are past the stage that they're producing random outputs of strings. Frontier models can perform an imitation of reasoning but the hallucination aspect seems to be more towards an inability to learn past it's training data or properly update it's neural net learnings when new evidence is presented. H…
Is that true? Is it anything more complicated than LLMs producing text optimized for plausibility rather than for any sort of ground version of truth?
Earlier quoted context omitted.
but you just described how to fix the "i don't know" problems to "i know and the answer is ". but not that "i don't know" is inherently hard to solve for some reason.
It's difficult to fix because the incentive is to make sure it has the answer, not to give it lots of questions to which there are known answers but have it answer "I don't know" (if you did that, you'd bias the model to be unable to answer those specific questions). Ergo, in inference, on questions not in the dataset, it's more inclined to make up an answer because it has very few "I don't know" samples in general.
> Other personality changes are subtler but still unsettling, like when models start sucking up to users or making up facts. My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement. The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function…
"I don't know" is one of possible answers. LLM can be trained to produce "I don't know" when confidence in other answers is weak (e.g. weak or mixed signals). Persona vector can also nudge it into that direction.
I'm unaware of -- and would love to find some -- convincing studies showing that LLMs have any kind of internal confidence metric. The closest I've seen is reflective chain-of-thought after the fact, and then trying to use per-token selection scores, which is doomed to fail (see: https://vlmsarebiased.github.io/)
Earlier quoted context omitted.
Sucking up does appear to be a personality trait. Hallucinations are not a completely known or well understood yet. We are past the stage that they're producing random outputs of strings. Frontier models can perform an imitation of reasoning but the hallucination aspect seems to be more towards an inability to learn past it's training data or properly update it's neural net learnings when new evidence is presented. H…
> Hallucinations are not a completely known or well understood yet. Is that true? Is it anything more complicated than LLMs producing text optimized for plausibility rather than for any sort of ground version of truth?
In the end, is driven towards engagement and delivering an answer and that drives it towards generating false answers when it doesn't know or understand.
If it was more personality controlled, delivering more humble and less confident answers or even making it say that it doesn't know would be a lot easier.
Earlier quoted context omitted.
That’s a really astute observation. It would be interesting if we could find a way to train models to signify when they are “stretching” the vector distance too far from the context window, because the available training data is too sparse or nonexistent. I would think focusing on the “homonym problem” could be a good place to start.
There is to my knowledge no vector signifying "truth" and therefore no vector to measure the distance from. You cannot get a "truthiness" measure out of these models, because they don't have the concept of truth. They use "likelyness" as a proxy for "truth". You could decide that the text is "too unlikely" the problem there is that you'll quickly discover that most human sentences are actually pretty unlikely.
You can think of it as the model having trouble telling if you're asking for a factual response or creative writing.