[dead]
Do LLMs pass the mirror test?
11–20 of 72 posts
Re: Do LLMs pass the mirror test?
#12Every LLM is a classifier biased towards its own writing, but the bias is usually subtle and the naive way like this is not reliable.
Re: Do LLMs pass the mirror test?
#13You can do much more, if you mess with harness, like translating model output language in realtime from english to french, or replacing some words.
If there is some sort of feedback loop (model has a reason to look into mirror), it usually does notice.
Re: Do LLMs pass the mirror test?
#14The styling on the website makes me feel like my phone is a cylinder
It's quite distracting and frustrating. No idea why you'd want the beginning and ends of lines of text to be darker than the center.
Re: Do LLMs pass the mirror test?
#15The styling on the website makes me feel like my phone is a cylinder
It's quite distracting and frustrating. No idea why you'd want the beginning and ends of lines of text to be darker than the center.
Sorry about that, the vignette was mainly meant for the desktop view only but is indeed much more invasive/disruptive in the mobile layout.
Should be better now.
Re: Do LLMs pass the mirror test?
#16A more appropriate mirror test for LLMs is to get them to state facts about their training data. Percentage of arts vs science for example. Given the framing that they're similar to nukes and a national security issue, it's likely that the models are post trained to not answer such questions accurately. Also the article could be trying to normalize thinking that these are more than matrix multiplication gadgets good…
It's not really "trying" to do anything. That they're, inherently, sequential matrix multipliers with clever data propagation should be uncontroversial, but I think stopping there is overly reductive.
Mechanistic interpretability research has found plenty of indicators that real, complex, generalized, and reusable circuits develop in models as they are trained and post-trained, particularly as overtraining ratios increase and memorization shifts to generalization. That's not to say that means they must be "conscious," but the overall point is that claiming anything definitive either way is incomplete.
It can be fascinating reading if you can sort through the chuff.
Re: Do LLMs pass the mirror test?
#17A more appropriate mirror test for LLMs is to get them to state facts about their training data. Percentage of arts vs science for example. Given the framing that they're similar to nukes and a national security issue, it's likely that the models are post trained to not answer such questions accurately. Also the article could be trying to normalize thinking that these are more than matrix multiplication gadgets good…
>Also the article could be trying to normalize thinking that these are more than matrix multiplication gadgets good at compression.
Honestly, I think it's less so (for some of us) that we think they're "more than matrix multiplication gadgets good at compression", so much as thinking that perhaps what our brains are doing is not so dissimilar.
A materialist view of the world could support the idea that intelligence itself may just be a series of predictions from a big compressed multi-modal dataset. That's not to say that LLMs are doing it in a way that is even close to how our brains are doing it, but we also don't understand how different it may be, and how much utility we can get out of them even with the current architecture.
Re: Do LLMs pass the mirror test?
#18It seems like we forget that LLMs are next token prediction systems. Using raw models without instruction following and chat completion bells and whistles will give you a better feeling of what LLMs are.
The current interface to LLMs are heavily biased towards "predict the next token in the context of a user with a helpful assistant" but LLMs are capable of other modes of next token prediction too.
Before the ChatGPT release people often measured LLM performance by how well they could produce a coherent story or a poem. that's where Anthropic model names are originating from I am guessing.
Re: Do LLMs pass the mirror test?
#19Why are we asking a language model for a mirror test? Just because it speak like human, have we forget what it is?
Re: Do LLMs pass the mirror test?
#20> The result was that dogs weren't interested in their unmodified scent in "raw" form, but the modified version was by far the most interesting thing in the room. They spent more time investigating it than any other stimulus in the experiment.
I know very well that this is kind of off-topic, and just like the author, i do not claim to know wether dogs (or any other non-human animal for that matter) is self-aware, and again, just like the author, i do think that the question cannot be answered. Either way, the modified version of their scent seemed more interesting to the dogs, maybe it's because they smell their own scent all the time. The single fact that their modified scent is more interesting to them does not mean they are self-aware, perhaps they are just trying something new.