Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Language models can explain neurons in language models

openai.com

Post reply on HN
Print view View original

41–50 of 497 posts

Re: Language models can explain neurons in language models

#41

Post by superkuh »

simonw wrote: post #11
cschmid wrote: post #4

Has anyone here found a link to the actual paper? If I click on 'paper', I only see what seems to be an awkward HTML version.

You mean this? https://openaipublic.blob.core.windows.net/neuron-explainer/... Would you prefer a PDF? (I'm always fascinated to hear from people who would rather read a PDF than a web-native paper like this one, especially given that web papers are actually readable on mobile devices. Do you do all of your reading on a laptop?)

With a pdf I don't have to update my PDF reader multiple times per month just to be able to read text.

A PDF is a text document that includes all the text, images, etc within it in the state you are going to perceive them. That web page is just barely even a document. None of it's contents are natively within it, it all requires executing remote code which pulls down more remote code to run just to get the actual text and images to display... which they don't in my browser. I just see an index with links that don't work and the the "Contributions" which for some reason was actually included as text.

Even as the web goes up it's own asshole in terms of recursive serial loading of javascript/json/whatever from unrelated domains and abandons all backwards compatibility, PDF, as a document, remains readable. I wish the web was still hyperlinked documents. The "application" web sucks for accessibility.

Re: Language models can explain neurons in language models

#42

Post by [deleted] »

jablongo wrote: post #18

This isnt exactly building an understanding of LLMs from first principles... IMO we should broadly be following the (imperfect) example set forth by neuroscientists attempting to explain fMRI scans and assigning functionality to various subregions in the brain. It is circular and "unsafe" from an alignment perspective to use a complex model to understand the internals of a simpler model; in order to understand GPT4 t…

[deleted]

Re: Language models can explain neurons in language models

#43

Post by drdeca »

mrcode007 wrote: post #19
fnovd wrote: post #6

LLMs are quickly going to be able to start explaining their own thought processes better than any human can explain their own. I wonder how many new words we will come up with to describe concepts (or "node-activating clusters of meaning") that the AI finds salient that we don't yet have a singular word for. Or, for that matter, how many of those concepts we will find meaningful at all. What will this teach us about…

If the Gödel incompleteness theorem applies here, then the explanations are likely … incomplete or self-referential.

What leads you to suspect that Gödel incompleteness may be relevant here?

There's no formal axiom system being dealt with here, afaict?

Do you just generally mean "there may be some kind of self-reference, which may lead to some kind of liar-paradox-related issues"?

Re: Language models can explain neurons in language models

#44

Post by fnovd »

chrisco255 wrote: post #30
fnovd wrote: post #6

LLMs are quickly going to be able to start explaining their own thought processes better than any human can explain their own. I wonder how many new words we will come up with to describe concepts (or "node-activating clusters of meaning") that the AI finds salient that we don't yet have a singular word for. Or, for that matter, how many of those concepts we will find meaningful at all. What will this teach us about…

Are there any examples of an LLM developing concepts that do not exist or cannot be inferred from its training set?

"Cannot be inferred from its training set" is a pretty difficult hurdle. Human beings can infer patterns that aren't there, and we typically call those hallucinations or even psychoses. On the other hand, some unconfirmed, novel patterns that humans infer actually represent groundbreaking discoveries, like for example much of the work of Ramanujan.

In a real sense, all of the future discoveries of mathematics already exist in the "training set" of our present understanding, we just haven't thought it all the way through yet. If we discover something new, can we say that the concept didn't exist, or that it "couldn't be inferred" from previous work?

I think the same would apply to LLMs and their understanding of the way we encode information using language. Given their radically different approach to understanding the same medium, they are well poised to both confirm many things we understand intuitively as well as expose the shortcomings of our human-centric model of understanding.

Re: Language models can explain neurons in language models

#45

Post by ftxbro »

I wonder will someone please check the neurons associated to the petertodd and other anomalous glitch tokens (https://www.lesswrong.com/posts/jkY6QdCfAXHJk3kea/the-petert...)? I can see the github and I see that for any given neuron you can see associated tokens but I don't see how to do an inverse search.

Re: Language models can explain neurons in language models

#46

Post by samgriesemer »

xthetrfd wrote: post #31

This blog post is not very informative. How did they prompt GPT4 to explain the neuron's behavior?

It's explained more in the "read paper" link, where they provide the actual prompts:

https://openaipublic.blob.core.windows.net/neuron-explainer/...

Re: Language models can explain neurons in language models

#47

Post by m1el »

srajabi wrote: post #3

"This work is part of the third pillar of our approach to alignment research: we want to automate the alignment research work itself. A promising aspect of this approach is that it scales with the pace of AI development. As future models become increasingly intelligent and helpful as assistants, we will find better explanations." On first look this is genius but it seems pretty tautological in a way. How do we know i…

You're correct to have a suspicion here. Hypothetically the explainer could omit a neuron or give a wrong explanation for the role of a neuron. Imagine you're trying to understand a neural network, and you spend enormous amount of time generating hypotheses and validating them. Well the explainer might give you 90% correct hypotheses, it means you have 10 times less work to produce hypotheses. So if you have a solid way of testing an explanation, even if the explainer is evil, it's still useful.

Re: Language models can explain neurons in language models

#48

Post by ShamelessC »

ftxbro wrote: post #2

> "This work is part of the third pillar of our approach to alignment research: we want to automate the alignment research work itself." I feel like this isn't a Yud-approved approach to AI alignment.

You mean Yudkowski? I saw him on Lex Fridman and he was entirely unconvincing. Why is everyone deferring to a bunch of effective altruism advocates when it comes to AI safety?

Re: Language models can explain neurons in language models

#50

Post by teaearlgraycold »

jacooper wrote: post #34

For people overwhelmed by all the AI science speak, just spend a few minutes with bing or phind and it will explain everything surprisingly well. Imagine telling someone in the middle of 2020, that in three years a computer will be able to speak, reason and explain everything as if it was a human, absolutely incredible!

I agree it’s crazy good. But timeline-wise, GPT-3 was in beta and used by many companies in 2020.
Post reply on HN