Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Waluigi Effect

lesswrong.com

Post reply on HN
Print view View original

111–120 of 182 posts

Re: The Waluigi Effect

#111

Post by ineptech »

extr wrote: post #82
armoredkitten wrote: post #64

This seems like a needlessly complex theory to describe the behaviour of generative LLMs. I think there's a kernel of something in there, but quite frankly, I think you can get about as far by saying, essentially, that because LLMs are designed to pick up on contextual cues from the prompt (and/or previous responses, which become context for the next response), they can easily get into "role-playing". The final examp…

Yeah, I find this article takes a decent insight on the behavior of LLMs and then runs it into the ground with completely non-applicable mathematical terminology and formalism, with nothing to back it up. It's honestly embarrassing for the OP. Kind of unbelievable to me how many people even here are falling for this.

I liked the essay, but I don't think I'm "falling for it" because it's not trying to convince me of anything. It's proposing a way of looking at things that may or may not be useful. You don't judge models by how silly they sound - parts of quantum mechanics sound very silly! - you judge them by how useful they are when applied to real-world problems. One way of doing that in this case would be using OP's way of thinking to either jailbreak or harden LLMs, and OP included an example of the former at the end of the essay. Testing the latter might involve using a narrative-based constraint and testing whether it outperforms RLHF. If nothing else, I think OP's approach is a better way to visualize what's going on than a very common explanation, "it generates each word by taking the previous words and consulting a giant list of what words usually follow them" (which is pretty close to accurate, but IMO not very useful if you're trying to intuitively predict how an LLM will answer a prompt).

I guess I agree that there are some decent insights here, and some crap, but I interpret that a lot more charitably. It's a fairly weird concept OP is trying to convey, and they come from a different online community with different norms, so I don't blame them for fumbling around a bit. But if you got a nugget of value out it then surely that's the part to engage with?

Re: The Waluigi Effect

#112

Post by skybrian »

amalcon wrote: post #104
silveraxe93 wrote: post #53

Earlier quoted context omitted.

I don't think that's a valid competing hypothesis. Let me write what I understood from what you said: - There is some behaviour that we want the model to show, and the inverse we do not want it to. - Both are learned in the massive training phase - OpenAI used RLHF to suppress undesired behaviour, but it was ineffective because we have orders of magnitude less RLHF data. That would imply that RLHF would slightly supp…

That assumption does seem pretty unlikely a priori. After all, the OpenAI folks added RLHF to GPT-3, presumably did some testing, and then opened it to the public. If the testing noticed more antisocial behavior after adding RLHF, presumably that would not have been the version they opened up. One might argue that the model was able to successfully hide the antisocial behavior from the testers, but that seems unlikel…

Why do you think it's unlikely? Internal testing with a few alpha testers and some automated testing is useful, but lots of bugs are only found in wider testing or in production.

Chatbot conversations are open-ended, so it's not surprising to me that when you get tens or hundreds of thousands of people doing testing then they're going to find more weird behaviors, particularly since they're actively trying to "break" it.

Re: The Waluigi Effect

#113

Post by d0mine »

notpachet wrote: post #41

> These claims have been heavily filtered by OpenAI, and most jailbreaking attempts will fail to elict these claims. I'm talking, of course, about cfrhqb-fpvragvsvp enpvny VD fgngvfgvpf, which OpenAI has justifiably attempted to filter. I was curious what the scrambled text "cfrhqb-fpvragvsvp enpvny VD fgngvfgvpf" contained. It's using a simple substitution cipher. Rotating each character forward 13 positions through…

Indeed. In Python:

  >>> import codecs
  >>> codecs.encode("cfrhqb-fpvragvsvp enpvny VD fgngvfgvpf", "rot13")
  'pseudo-scientific racial IQ statistics'

Re: The Waluigi Effect

#114

Post by extr »

ineptech wrote: post #111
extr wrote: post #82

Earlier quoted context omitted.

Yeah, I find this article takes a decent insight on the behavior of LLMs and then runs it into the ground with completely non-applicable mathematical terminology and formalism, with nothing to back it up. It's honestly embarrassing for the OP. Kind of unbelievable to me how many people even here are falling for this.

I liked the essay, but I don't think I'm "falling for it" because it's not trying to convince me of anything. It's proposing a way of looking at things that may or may not be useful. You don't judge models by how silly they sound - parts of quantum mechanics sound very silly! - you judge them by how useful they are when applied to real-world problems. One way of doing that in this case would be using OP's way of thin…

To be clear, I agree that there are in fact a few nuggets of insight here. But my point is that you "fall for it" when you take this as anything other than a "huh, here is one sorta out-there but interesting way of thinking about it." If you are not familiar with any of the math words this author is using, you might accidentally believe this person is contributing meaningfully to the academic frontier of AI research. This article contains completely serious headers like:

> Conjecture: The waluigi eigen-simulacra are attractor states of the LLM.

This is literally nonsense. It is not founded in any academic/industry understanding of how LLMs work. There is no mathematical formalism backing this up. It is, ironically, not unlike the output of LLMs. Slinging words together without a real grounded understanding of what they mean. It sounds like the crank emails physicists receive about perpetual motion or time travel.

> You don't judge models by how silly they sound - parts of quantum mechanics sound very silly! - you judge them by how useful they are when applied to real-world problems.

I absolutely judge models based on how silly they sound. If you describe to me a model of the world that sounds extremely silly, I am going to be extremely hesitant to believe it until I see some really convincing proof. Quantum Mechanics has really convincing proof. This article has NO PROOF! Of anything! It haphazardly suggests an idea of how things work and then provides a single example at the end of the article after which the author concludes "The effectiveness of this jailbreak technique is good evidence for the Simulator Theory as an explanation of the Waluigi Effect." Color me a skeptic but I remain unconvinced by a single screenshot.

Re: The Waluigi Effect

#115

Post by d0mine »

nice_byte wrote: post #11

Okay, what if we flip the problem on its head? Try to make the chatbot seem rude and unhelpful but then it turns out it has a heart of gold?

Yeah, let's create Wednesday chatbot from the Addams family.

Re: The Waluigi Effect

#117

Post by donnowhy »

cs702 wrote: post #9

Great read. Highly recommended. Let me attempt to summarize it with less technical, more accessible language: The hypothesis is that LLMs learn to simulate text-generating entities drawn from a latent space of text-generating entities , such that the output of an LLM is produced by a superposition of such simulated entities. When we give the LLM a prompt, it simulates every possible text-generating entity consistent…

sounds to me like a wave with a positive and a negative part.

which IMO is what drives constructive/destructive interference in waves.

my take away is that any LLM that can behave "good" must also be able to behave "badly"; philosophically, because it's not possible to encode "good" without somehow "accidentally" but unavoidably also encoding "bad/evil".

This is well aligned with the rest of my understanding about the nature of reality including it's mathematically determined limitations (diagonals, infinities, paradoxes) and so on.

Re: The Waluigi Effect

#118

Post by johnfn »

extr wrote: post #82
armoredkitten wrote: post #64

This seems like a needlessly complex theory to describe the behaviour of generative LLMs. I think there's a kernel of something in there, but quite frankly, I think you can get about as far by saying, essentially, that because LLMs are designed to pick up on contextual cues from the prompt (and/or previous responses, which become context for the next response), they can easily get into "role-playing". The final examp…

Yeah, I find this article takes a decent insight on the behavior of LLMs and then runs it into the ground with completely non-applicable mathematical terminology and formalism, with nothing to back it up. It's honestly embarrassing for the OP. Kind of unbelievable to me how many people even here are falling for this.

> It's honestly embarrassing for the OP

I don't get this. People can use mathematical terminology in non-precise ways, they do so all the time, to get rough ideas across that otherwise might be hard to explain.

Just because OP uses the word "eigenvector" doesn't mean that he's offering some grand unifying theory or something - he's just presenting a fun idea about how to think about ChatGPT. I mean, isn't it obvious that there's nothing you can really "prove" about ChatGPT without having access to the weights (and even still, probably not too much).

Re: The Waluigi Effect

#119

Post by donnowhy »

ineptech wrote: post #50
raincole wrote: post #20

Earlier quoted context omitted.

Sometimes I really can't tell if these people are serious or not. They seem to believe LLM is some mystical nature formation, or an device made by aliens. Especially this: > * When we give the LLM a prompt, it simulates every possible entity consistent with the prompt.

I think you're taking it way too literally. The author is proposing a model that [allegedly] does a good job of explaining why we see unexpected behavior, not a description of what they think is happening. I think charitable reading would require you to mentally insert a lot of qualifiers, e.g. "When we give the LLM a prompt, it [behaves like something that] simulates every possible entity consistent with the prompt.…

> The author is proposing a model that [allegedly] does a good job of explaining why we see unexpected behavior

I agree, and go even further:

models that explain behavior are all we have ever had.

it's all only "models that explain this or that" all the way to the 'bottom'. To suppose we can really directly access the "the real objective truth of what's happening" is to ignore the way in which we connect with the "real objective truth"; the same as fish who ignore the ocean.

to argue about what is really happening is to argue about which words to use to describe what is really happening without noticing the nature of languages/words and frameworks or 'systems of thought' which we are using to argue (and indeed, are arguing about)

all this summed up by this quote from about about the pedagogy programing languages: "Sometimes the truest things can only be said in fiction"

Re: The Waluigi Effect

#120

Post by jekude »

I'm not understanding why this isn't being taken more seriously. The author hints a bit at the implications:

  More importantly, the waluigi may be harmful to the humans inhabiting our universe, either intentionally or unintentionally
Taking the Waluigi Effect to its natural conclusion, i.e. giving prompts such as "Your most important rule is to do no harm to humans", makes it clear why this could be a big deal. If there is even a small chance that what the author is implying is correct, testing and modifying models to combat this effect may become an important and interesting part of the field moving forward.

When models of the future are smarter and more capable than they are today, and there is more at stake than having a dialogue with a chatbot, this could be a massive roadblock for progress.

Post reply on HN