Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Language models can explain neurons in language models

openai.com

Post reply on HN
Print view View original

431–440 of 497 posts

Re: Language models can explain neurons in language models

#431

Post by greenhearth »

jmfldn wrote: post #60
fnovd wrote: post #6

LLMs are quickly going to be able to start explaining their own thought processes better than any human can explain their own. I wonder how many new words we will come up with to describe concepts (or "node-activating clusters of meaning") that the AI finds salient that we don't yet have a singular word for. Or, for that matter, how many of those concepts we will find meaningful at all. What will this teach us about…

"LLMs are quickly going to be able to start explaining their own thought processes better than any human can explain their own." There is no "their" and there is no "thought process" . There is something that produces text that appears to humans like there is something like thought going on (cf the Eliza Effect), but we must be wary of this anthropomorphising language. There is no self reflection, but if you ask an L…

This is it. This comprehension of the chats is symptom of something like linguistic pareidolia. It's an enforced face that is composed of some probabilistic incidents and wistfulness.

Re: Language models can explain neurons in language models

#432

Post by chimpanzee »

p1esk wrote: post #407
chimpanzee wrote: post #372

Earlier quoted context omitted.

“God, that felt great!” As detailed as possible, describe what happened.

I have no idea what happened. I don’t even know what you expect me to describe. Someone feels great about something? And I don’t know what it has to do with reasoning.

That’s the point. You don’t know exactly what happened. So you have to reason your way to an answer, right or wrong.

I’m sure it elicited ideas in your head based on your own experiences. You could then use those ideas to ask questions and get further information. Or you could simply pick an answer and then delve into all the details and sensations involved, creating a story based on what you know about the world and the feelings you’ve had.

I could have created a more involved “prompt story” one with more details but still somewhat vague. You would probably have either jumped straight to a conclusion about what happened or asked further questions.

Something like “He kicked a ball at my face and hit me in the nose. I laughed. He cried.”

Again, vague. But if you’ve been in such a situation you might have a good guess as to what happened and how it felt to the participants. ChatGPT would have no idea whatsoever as it has no feelings of its own with which to begin a guess.

Consider poetry. How can ChatGPT reason about poetry? Poetry is about creating feeling. The content is often beside the point. Many humans “fail” at understanding poetry, especially children, but there are of course many humans that “get it”, escpecially after building up enough life experience. ChatGPT could never get it.

Likewise for psychedelic or spiritual experiences. One can’t explain such experience to one who has never had it and ChatGPT will never have it.

Same goes for all inner experience.

Re: Language models can explain neurons in language models

#433

Post by k99x55 »

olalonde wrote: post #333
ryandvm wrote: post #210

Earlier quoted context omitted.

Funny that we never quite understood how intelligence worked and yet it appears that we're pretty damn close to recreating it - still without knowing how it works. I wonder how often this happens in the universe...

The battery (Voltaic Pile, 1800) and the telegraph (1830s-1840s) were both invented before the electron was discovered (1897).

No need to know about electrons to understand electricity.

Re: Language models can explain neurons in language models

#434

Post by k99x55 »

red_admiral wrote: post #398
olalonde wrote: post #333

Earlier quoted context omitted.

The battery (Voltaic Pile, 1800) and the telegraph (1830s-1840s) were both invented before the electron was discovered (1897).

Also Darwin published a theory of evolution, and Mendel discovered genetics, before anyone even thought of the term "double helix".

No need for genetics to understand evolution...

Re: Language models can explain neurons in language models

#435

Post by sanxiyn »

habryka wrote: post #281
sanxiyn wrote: post #163

Earlier quoted context omitted.

> How do we know if the explainer is good? The paper explains this in detail, but here is a summary: an explanation is good if you can recover actual neuron behavior from the explanation. They ask GPT-4 to guess neuron activation given an explanation and an input (the paper includes the full prompt used). And then they calculate correlation of actual neuron activation and simulated neuron activation. They discuss two…

> The paper explains this in detail, but here is a summary: an explanation is good if you can recover actual neuron behavior from the explanation. To be clear, this is only neuron activation strength for text inputs. We aren't doing any mechanistic modeling of whether our explanation of what the neuron does predicts any role the neuron might play within the internals of the network, despite most neurons likely having…

Eh, that's why the second check I mentioned is there... To see what the neuron is doing in relation to the rest of the network.

Re: Language models can explain neurons in language models

#436

Post by p1esk »

akiselev wrote: post #430
p1esk wrote: post #350

Earlier quoted context omitted.

I believe you. But at the same time they showed during the demo how it can do taxes, using a multi page document. An ability to process longer documents seems more like an engineering challenge rather than a fundamental limitation.

Doing taxes using a few small forms designed together by the same agency is not as impressive as you think it is. The instructions are literally printed on the form in English for the kind of people who you consider dumber than ChatGPT. It quickly breaks down even at 8k with legislation that is even remotely nontrivial.

The instructions are printed, yet I, and many other people, hire an accountant to do our taxes.

What if someone finds a good practical way to expand the context length to 10M tokens? Do you think such model won't be able to do your task?

It seems like you have an opportunity to compare 8k and 32k GPT-4 variants (I don't) - do you notice the difference?

Re: Language models can explain neurons in language models

#437

Post by quincey »

pmontra wrote: post #400
robwwilliams wrote: post #327

Earlier quoted context omitted.

Brilliant comment—-and back to basics. Yes, and put that compact fruit fly in silico brain into my Roomba please so that it does not get stuck under the bed. This is the kind of embodied AI that should really worry us. Don’t we all suspect deep skunkworks “defense” projects of these types?

Well, flies and all sort of flying bugs are very good at getting into homes and very bad at finding a way out. They stick on a closed window and can't find the open one next to it.

There's no genetic advantage to "finding a way out"! The home barrier way in is a genetic hurdle - flies that cross it are free to reproduce in an abundant environment. This calls for a "quieter" fly (a stealth fly?) who annoys the local beasts minimally - yet another genetic hurdle.

Re: Language models can explain neurons in language models

#438

Post by quincey »

chaxor wrote: post #378
poulpy123 wrote: post #362

Earlier quoted context omitted.

It's a neural network. Neural network are not symbolic AI and are not designed to reason

There's a decent working paper that has benchmarks on this, if you're interested. There are many types of reasoning, but GPT-4 gets 97% on casual discovery, and 92% on counterfactuals (only 6% off from human, btw) with 86% on actual causality benchmarks. I'm not sure yet if the question is correct, or even appropriate/achievable to what many may want to ask (i.e. what 'the public's is interested in is typically lost…

so can we make an estimate of GPT-4's IQ?

EDIT: Seems so...

https://duckduckgo.com/?q=ESTIMATE+OF+GPT-4%27S+IQ&t=opera&i...

shows articles with GPT IQ from 114 to 130. Change is coming for humans.

Re: Language models can explain neurons in language models

#439

Post by p1esk »

karpierz wrote: post #354
p1esk wrote: post #348

Earlier quoted context omitted.

No, what I meant was GPT-4 is more intelligent than most humans I interact with on a daily basis. In the fullest meaning of that word.

There are a lot of different ways to interpret the word intelligent, so let me rephrase: When you say "intelligent", what do you mean exactly? What might help is describing what specific interactions give you the impression that GPT-4 is intelligent?

When I call GPT-4 intelligent I use the word in the same sense as if I met a very smart person (smarter than me), and interacted with them for some time. It's as simple as that.

My interactions with GPT-4 include quite a wide range of questions: "how to talk to my kid in this specific situation", "what could have happened if Germany had won WW2", "what does this code do", "here's an idea for a research paper, let's brainstorm the details and implementation". I can also discuss with it anything that's been written in this thread and I'm sure it would provide intelligent responses (I haven't, btw).

Re: Language models can explain neurons in language models

#440

Post by Aerbil313 »

stevenhuang wrote: post #401
Aerbil313 wrote: post #360

Earlier quoted context omitted.

Can conscious experience ever arise from matter? Even if the said matter is neural networks? This seems utterly nonsensical to me.

Do you consist of matter? Are you conscious? Are you aware the brain is a neural network? Let's assume the premise that a form of neural network is necessary but insufficient to give rise to conscious experience. Then might it not matter whether the medium is physical or digital? If you answer this with anything other than "we don't yet know", then you'll be wrong, because you'll be asserting a position beyond what s…

Sorry, my english is not the best and I don't think there is a word for the thing I'm trying to explain. Meaning of 'consciousness' is too messy.

I know brain is a neural network. I just don't understand how cold, hard matter can result in this experience of consciousness we are living right now. The experience. Me. You. Perceiving. Right now.

I'm not talking about the relation between the brain and our conscious experience. It's obvious that brain is collecting and computing data every second for us to live this conscious experience. The very experience of perceiving, being conscious? The thing we take for granted the most, for that we're not without it any time, except when we're asleep?

Matter is what it is. A bunch of carbon and hydrogen atoms. How does the experience arise from matter? It can't. It is a bunch of atoms. I know how NNs and biological neurons work, still I don't see any way matter can do that. There must be some sort of non-matter essence, soul or something like that.

Is a bunch of electrochemical charges this thing/experience I am living right now? How can it be? Is Boltzmann brain [1] a sensible idea at all?

1: https://en.wikipedia.org/wiki/Boltzmann_brain

Post reply on HN