> If you ask GPT-
∞
"what's brown and sticky?", then it will reply "a stick", even though a stick isn't actually sticky.
Isn't it though?
The Waluigi Effect
31–40 of 182 posts
Re: The Waluigi Effect
#32Earlier quoted context omitted.
Superposition just means “linear combination” in this context. Basically, a weighted mixture of “simulated entities” (or possible responses). https://en.m.wikipedia.org/wiki/Superposition_principle
The author liberally alludes to “superposition collapse,” which implies that they’re referring to its quantum mechanical meaning.
It doesn't imply that. What term would you use to refer to the narrowing of a hypothesis space upon acquisition of new evidence?
Re: The Waluigi Effect
#33> If you ask GPT- ∞ "what's brown and sticky?", then it will reply "a stick", even though a stick isn't actually sticky. Isn't it though?
Also, isn't this a really common joke? I assume ChatGPT will have absorbed some amount of a sense of humor from its trawls of the internet.
Re: The Waluigi Effect
#34Great read. Highly recommended. Let me attempt to summarize it with less technical, more accessible language: The hypothesis is that LLMs learn to simulate text-generating entities drawn from a latent space of text-generating entities , such that the output of an LLM is produced by a superposition of such simulated entities. When we give the LLM a prompt, it simulates every possible text-generating entity consistent…
Sometimes I really can't tell if these people are serious or not. They seem to believe LLM is some mystical nature formation, or an device made by aliens. Especially this: > * When we give the LLM a prompt, it simulates every possible entity consistent with the prompt.
Would you object if the sentence read "it approximates simulating every possible entity consistent with the prompt"?
Re: The Waluigi Effect
#35Earlier quoted context omitted.
The author liberally alludes to “superposition collapse,” which implies that they’re referring to its quantum mechanical meaning.
It doesn't imply that. What term would you use to refer to the narrowing of a hypothesis space upon acquisition of new evidence?
Bayesian inference?
Re: The Waluigi Effect
#36Earlier quoted context omitted.
The author liberally alludes to “superposition collapse,” which implies that they’re referring to its quantum mechanical meaning.
It doesn't imply that. What term would you use to refer to the narrowing of a hypothesis space upon acquisition of new evidence?
In a non-Bayesian context, I would call it “updating/retraining my model.”
In a formal Bayesian context, I’d call it “updating my posterior by adding data to the likelihood.”
Re: The Waluigi Effect
#37This is fun to read and think about, but it's also important to keep in mind that this is very light on evidence and is basically fanfic. The fact that the author uses entertaining Waluigi memes shouldn't convince you that it's true. LessWrong has a lot of these types of posts that get traction because they're much heavier on memes than experiments and data.
Here is a competing hypothesis:
The capability to express so-called Waluigi behavior emerges from the general language modeling task. This is where the vast majority of information is - it's billions or even trillions of tokens with token-level self-supervision. All of the capabilities are gained here. RLHF has a tiny amount of information by comparison - it's just a small amount of human-ranked completions. It doesn't even train with humans "in the loop", their rankings are acquired off-line and used to train a weak preference model. RLHF doesn't have enough information to create a "Luigi" or a "Waluigi", it's just promoting pre-existing capabilities. The reason you can get "Waluigi" behavior isn't because you tried to create a Luigi. It's because that behavior is already in the model from the language modeling phase. You could've just as easily elicited Waluigi responses from the pure language model before RLHF.
There's no super-deceptive Waluigi simulacra that's fooling human labelers into promoting it during RLHF - this should be obvious from the fact that we can immediately identify the undesirable behavior of Bing.
Re: The Waluigi Effect
#38A few weeks ago when people were speculating as to why Microsoft's chatbot went feral, one explanation people were converging on is that the space of all human writing ever produced, being a collective production of the human psyche, contains several attractor states corresponding to human personality archetypes, and that Microsoft's particular (probably rushed) RLHF training operation had landed Sydney in the "neuro…
Don’t let the mathematical terms fool you, these are just fan theories. For a real investigation you need debug access. For a good example:
https://clementneo.com/posts/2023/02/11/we-found-an-neuron
For image recognition, machine learning researchers eventually figured out the neural networks are paying attention mostly to textures. Hopefully we will have a better understanding of what language models are doing someday.
Re: The Waluigi Effect
#39I would question the assumption that there is a simulacrum of anything in a LLM, not even implicit. Any simulacrum, identity, self-consistency etc. is a projection of the "reader", i.e. user.
(I guess it is an interesting philosophical question whether a convincing presentation of a simulation of a mind is a mind, or at least an acceptable simulation. One meta level higher as the turing test, so to speak. If so, I'm not sure we are ready to accept the consequences. For example, that a "chinese room" would be just as concious as a human, or a human just as non-concious as a LLM.)
What's actually going on is that a LLM is like the language center of a brain, without the brain. Especially there is no world-model, and no inner state. It just goes by context, which works most of the time, but as pointed out that is not enough and it can flip "personalities". The article is still really interesting, because it is a neat way to think about these models. I think it is absolutely not what is going on, but it is a good intuitive model why prompts go bad.
Re: The Waluigi Effect
#40Earlier quoted context omitted.
It doesn't imply that. What term would you use to refer to the narrowing of a hypothesis space upon acquisition of new evidence?
In a non-Bayesian context, I would call it “updating/retraining my model.” In a formal Bayesian context, I’d call it “updating my posterior by adding data to the likelihood.”
That could describe both the narrowing or broadening of one's hypothesis space.