Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Waluigi Effect

lesswrong.com

Post reply on HN
Print view View original

71–80 of 182 posts

Re: The Waluigi Effect

#71

Post by [deleted] »

ineptech wrote: post #50
raincole wrote: post #20

Earlier quoted context omitted.

Sometimes I really can't tell if these people are serious or not. They seem to believe LLM is some mystical nature formation, or an device made by aliens. Especially this: > * When we give the LLM a prompt, it simulates every possible entity consistent with the prompt.

I think you're taking it way too literally. The author is proposing a model that [allegedly] does a good job of explaining why we see unexpected behavior, not a description of what they think is happening. I think charitable reading would require you to mentally insert a lot of qualifiers, e.g. "When we give the LLM a prompt, it [behaves like something that] simulates every possible entity consistent with the prompt.…

[deleted]

Re: The Waluigi Effect

#72

Post by assbuttbuttass »

raincole wrote: post #20
cs702 wrote: post #9

Great read. Highly recommended. Let me attempt to summarize it with less technical, more accessible language: The hypothesis is that LLMs learn to simulate text-generating entities drawn from a latent space of text-generating entities , such that the output of an LLM is produced by a superposition of such simulated entities. When we give the LLM a prompt, it simulates every possible text-generating entity consistent…

Sometimes I really can't tell if these people are serious or not. They seem to believe LLM is some mystical nature formation, or an device made by aliens. Especially this: > * When we give the LLM a prompt, it simulates every possible entity consistent with the prompt.

> When we give the LLM a prompt, it simulates every possible entity consistent with the prompt.

This is just Bayes' rule. The probability of an LLM generating any particluar output is the sum over every possible entity of the probability of seeing that entity multiplied by the chance that entity would generate the output.

Re: The Waluigi Effect

#74

Post by lukeplato »

armoredkitten wrote: post #64

This seems like a needlessly complex theory to describe the behaviour of generative LLMs. I think there's a kernel of something in there, but quite frankly, I think you can get about as far by saying, essentially, that because LLMs are designed to pick up on contextual cues from the prompt (and/or previous responses, which become context for the next response), they can easily get into "role-playing". The final examp…

they are specifically pointing out that the process of RLHF, which is intended to add guard rails on the chat bots trajectory through an all encompassing latent space of internet data, has an unintentional side-effect of creating a highly characterized alter-ego that can more easily be summoned.

The theory is well-thought-out and necessarily rich. The psychological approach of analysis from the alignment crowd is much overdue.

Re: The Waluigi Effect

#75

Post by Rebelgecko »

>a reply to a question is more likely to be correct when the character has already been described as a smart, honest, helpful, harmless, etc.

Is that actually true? FWIW I've often ran into the reddit equivalent of Gell-Mann amnesia. In a thread about some niche topic I'm fairly knowledgeable about (something I've worked on professionally for years where there's maybe 10k people globally who know it better than I do), I post a comment that gets downvoted to hell, while there's a highly upvoted comment from someone who clearly just skimmed Wikipedia and poorly paraphrased the intro article.

Re: The Waluigi Effect

#76

Post by exitb »

captainmuon wrote: post #39

I would question the assumption that there is a simulacrum of anything in a LLM, not even implicit. Any simulacrum, identity, self-consistency etc. is a projection of the "reader", i.e. user. (I guess it is an interesting philosophical question whether a convincing presentation of a simulation of a mind is a mind, or at least an acceptable simulation. One meta level higher as the turing test, so to speak. If so, I'm…

> What's actually going on is that a LLM is like the language center of a brain, without the brain.

I’ve seen this sentiment expressed multiple times, but is that really correct? Maybe this works differently for other people, but I’ve noticed that I have to use my language to really think. I can do trivial things mindlessly, but to solve a problem, I need to express it with words in my mind. It makes me feel like the most important parts of the brain actually are fancy language models.

Re: The Waluigi Effect

#77

Post by sdwr »

ineptech wrote: post #50
raincole wrote: post #20

Earlier quoted context omitted.

Sometimes I really can't tell if these people are serious or not. They seem to believe LLM is some mystical nature formation, or an device made by aliens. Especially this: > * When we give the LLM a prompt, it simulates every possible entity consistent with the prompt.

I think you're taking it way too literally. The author is proposing a model that [allegedly] does a good job of explaining why we see unexpected behavior, not a description of what they think is happening. I think charitable reading would require you to mentally insert a lot of qualifiers, e.g. "When we give the LLM a prompt, it [behaves like something that] simulates every possible entity consistent with the prompt.…

As with all these articles, the title and main premise are misguided and/or imaginary, but there are some worthwhile nuggets.

The superposition of possible attitudes is a good one. Even if that's not the way LLMs "actually" work, it's descriptive of the possibility space from our perspective. And the dive into narrative theory + the stickiness of opposites is nice. Narratives have their own momentum in a "stone soup" kind of way - everyone who hears it participates and adds fuel to the fire. Even rejecting the narrative gives it validity in a price anchoring / overton window way.

Re: The Waluigi Effect

#78

Post by mitthrowaway2 »

attemptone wrote: post #48
mitthrowaway2 wrote: post #34

Earlier quoted context omitted.

Would you object if the sentence read "it approximates simulating every possible entity consistent with the prompt"?

Not OP, but I see the also in problem with 'every possible entity'. If you formulate it like that the prompt is decoupled from the LLM capabilities and can be anything. And if you restrict the prompt to cover only what the LLM understands the sentence becomes trivial. Train a LLM with ASCII and try to get it to simulate anything that is outside of that (ancient sumerian script for example). If you only input ASCII it…

I think "simulating" in this context means internally executing a process that is very similar to the process that generated the original material, as part of the prediction process. In general, that's the most compact way to predict and reproduce the original material.

For example, the string "1010101010"... could be the output of a function

  def generate_char_random(prev_string):
      x = random()
      if (x > 0.5):
         yield(1)
      else:
         yield(0)
It could also be the output of this function:

  def generate_char_alternating(prev_string):
      x = float(prev_string[-1])
      if (x
Even if it's not explicitly running those two functions, a model that is very good at predicting the next character of this input string might have, embedded within it, analogues of both of those two functions. The longer the output continues to follow the "101010" pattern, the higher confidence it should place on the _alternating version. On the other hand, if it encounters a "...110001..." sequence, it should switch to placing much more confidence on the _random version.

The LLM of course does not contain an infinite list of generative functions and weight their outputs. But to the extent that it works well and compactly approximates Bayesian reasoning, it should approximate a program that does.

Re: The Waluigi Effect

#79

Post by mola »

Article is proof that rationalism without empiricism is useless. It just drives to weird dead ends for no reason. Before theorizing for explanation of an effect check if the effect actually exist? Bah, this article is such a waste of computer memory

Re: The Waluigi Effect

#80

Post by zehaeva »

exitb wrote: post #76
captainmuon wrote: post #39

I would question the assumption that there is a simulacrum of anything in a LLM, not even implicit. Any simulacrum, identity, self-consistency etc. is a projection of the "reader", i.e. user. (I guess it is an interesting philosophical question whether a convincing presentation of a simulation of a mind is a mind, or at least an acceptable simulation. One meta level higher as the turing test, so to speak. If so, I'm…

> What's actually going on is that a LLM is like the language center of a brain, without the brain. I’ve seen this sentiment expressed multiple times, but is that really correct? Maybe this works differently for other people, but I’ve noticed that I have to use my language to really think. I can do trivial things mindlessly, but to solve a problem, I need to express it with words in my mind. It makes me feel like the…

I know I am in the minority out there but when I do math, calculus, diff eq, whatever, the answer just comes to me. There's no internal dialogue, the answer just, for the lack of a better phrase, rises from the deep and is known to me.

When I am in an discussion I will look up and off the my left when I am thinking, but no words are happening in my "inner dialogue", it's just nothing and then I start speaking whatever I paused for.

Similar things happen to me while I am programming at work, I stare at the problem and the answer just comes.

Post reply on HN