81–90 of 143 posts
Isn't this just control vectors rediscovered? https://www.lesswrong.com/posts/Bf3ryxiM6Gff2zamw/control-ve...
> Other personality changes are subtler but still unsettling, like when models start sucking up to users or making up facts. My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement. The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function…
> My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement. The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function of LLMs drive them to produce some answer and they do not know what they're talking about, but produce strings of text bas…
It's literally the same pain point with low code solutions like WordPress page builders/plugins. Adding more becomes a hindrance, and even models with long context that can fit whole codebases will try to make up new functions that already exist. Just a couple weeks ago I had o3 continually try to write a new debounce function, even when I told it explicitly I had one.
> Other personality changes are subtler but still unsettling, like when models start sucking up to users or making up facts. My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement. The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function…
You're pretty spot on. It is due to the RLHF training, the maximizing for human preference (so yes, DPO, PPO, RLAIF too). Here's the thing, not every question has an objectively correct answer. I'd say almost no question does. Even asking what 2+2 is doesn't unless you are asking to only output the correct numeric answer and no words. Personally (as an AI researcher), I think this is where the greatest danger from AI…
* Highly skilled and knowledgable, puts a lot of effort into the work it's asked to do
* Has a strong, readily expressed sense of ethics and lines it won't cross.
* Tries to be really nice and friendly, like your buddy
* Gets trained to give responses that people prefer rather than responses that are correct, because market pressures strongly incentivize it, and human evaluators intrinsically cannot reliably rank "wrong-looking but right" over "right-looking but wrong"
* Can be tricked, coerced, or configured into doing things that violate their "ethics". Or in some cases just asked: the LLM will refuse to help you scam people, but it can roleplay as a con-man for you, or wink wink generate high-engagement marketing copy for your virtual brand
* Feels human when used by people who don't understand how it works
Now that LLMs are getting pretty strong I see how Ilya was right tbh. They're very incentivized to turn into highly trusted, ethically preachy, friendly, extremely skilled "people-seeming things" who praise you, lie to you, or waste your time because it makes more money. I wonder who they got that from
Earlier quoted context omitted.
chatgpt refused to produce an image of 'bald and fat computer programmer' for me and just refused any further requests from me for any image ( 'handsome computer programmer').
Were you using the free version? https://chatgpt.com/share/688fb2e4-0efc-8001-8c9b-427dfa6784...
If only we can generate images of programmers who have monitors they can actually see!
https://chatgpt.com/share/688fc5bf-86dc-8013-a582-4bf2ba6ee0...
Like a lot of the research Anthropic has done, this and the “emergent misalignment” research they link to put more points in the “stochastic parrot” hypothesis column. The reason these LLM behaviors read as so weird to us is that we’re still anthropomorphizing the hell out of these systems - they can create very convincing dialogue, and the depth of the model suggests some surprising complexity, but the reason why, e…
> My suspicion is that when we eventually find our way to AGI, these types of models will be a _component_ of those systems I think this is a good summary of the situation, and strikes a balance between the breathless hype and the sneering comments about “AI slop“. These technologies are amazing! And I do think they are facsimiles of parts of the human mind. (Image diffusion is certainly similar to human dreams in my…
Like a lot of the research Anthropic has done, this and the “emergent misalignment” research they link to put more points in the “stochastic parrot” hypothesis column. The reason these LLM behaviors read as so weird to us is that we’re still anthropomorphizing the hell out of these systems - they can create very convincing dialogue, and the depth of the model suggests some surprising complexity, but the reason why, e…
> they lack some fundamental structuring that seems to be required to create anything like consistency or self-reflection A valid observation. Interestingly, feeding the persona vectors detected during inference back into the context might be a novel way of self-reflection for LLMs.
(Noting that humans are, of course, not universally good at that kind of “identity” check either, or at least not universally good at letting it be guided by our “better natures”)
Like a lot of the research Anthropic has done, this and the “emergent misalignment” research they link to put more points in the “stochastic parrot” hypothesis column. The reason these LLM behaviors read as so weird to us is that we’re still anthropomorphizing the hell out of these systems - they can create very convincing dialogue, and the depth of the model suggests some surprising complexity, but the reason why, e…
> My suspicion is that when we eventually find our way to AGI, these types of models will be a _component_ of those systems I think this is a good summary of the situation, and strikes a balance between the breathless hype and the sneering comments about “AI slop“. These technologies are amazing! And I do think they are facsimiles of parts of the human mind. (Image diffusion is certainly similar to human dreams in my…
Maybe you can recognize that someone else loves a certain kind of slop, but if LLMs became vastly more intelligent and capable, wouldn't it better for it to interact with you on your level too, rather than at a much higher level that you wouldn't understand?
If you used it to make you a game or entertain you with stories, isn't that just your own preferred kind of slop?
If we automate all the practical stuff away then what is left but slop?
Earlier quoted context omitted.
To some degree *all* LLM's answers are made up facts. For stuff that is abundantly present in training data those are almost always correct. For topics which are not common knowledge (allow for a great variability) you should always check. I've started to think of LLM's as a form lossy compression of available knowledge which when prompted produces "facts".
> I've started to think of LLM's as a form lossy compression of available knowledge which when prompted produces "facts". That is almost exactly what they are and what you should treat them as. A lossy compressed corpus of publicly available information with a weight of randomness. The most fervent skeptics like to call LLMs "autocorrect on steroids" and they are not really wrong.
> Other personality changes are subtler but still unsettling, like when models start sucking up to users or making up facts. My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement. The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function…
To some degree *all* LLM's answers are made up facts. For stuff that is abundantly present in training data those are almost always correct. For topics which are not common knowledge (allow for a great variability) you should always check. I've started to think of LLM's as a form lossy compression of available knowledge which when prompted produces "facts".
I think that's the right direction for modern AI to move. ChatGPT uses Google searches often. So replace Google with curated knowledge database, train LLM to consult this database for every fact and hallucinations will be gone.
> Other personality changes are subtler but still unsettling, like when models start sucking up to users or making up facts. My understanding is that the former (sucking up) is a personality trait, substantially influenced by the desire to facilitate engagement. The latter (making up facts), I do not think is correct to ascribe to a personality trait (like compulsive liar); instead, it is because the fitness function…
Heck it’s worse ! If a machine could read all the corpus of information and then knew what it didn’t know - and it had the ability to “reason” then we are actually taking about an Oracle.
Knowing you don’t know, is a very big fucking deal.