Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Language models can explain neurons in language models

openai.com

Post reply on HN
Print view View original

321–330 of 497 posts

Re: Language models can explain neurons in language models

#321

Post by chimprich »

benlivengood wrote: post #195
srcreigh wrote: post #182

Earlier quoted context omitted.

What's the argument that understanding neurons is necessary? Perhaps intelligence is like a black box input to our bodies (call it the "soul", even though this isn't testable and therefore not a hypothesis). The mind therefore wouldn't play any more of a role in intelligence than the eye. And I'm not sure people would say the eye is necessary for understanding intelligence. Now, I'm not really in a position to argue…

Brain damage by physical trauma, disease, oxygen deprivation, etc. has dramatic and often permanent effects on the mind. The effect of drugs (including alcohol) on the mind. Of note is anesthesia which can reliably and reversibly stop internal experience in the mind. For a non-physical soul to hold our mind we would expect significant divergence from the above. Out of body experiences and similar are indistinguishabl…

> Brain damage by physical trauma, disease, oxygen deprivation, etc. has dramatic and often permanent effects on the mind.

That's not a completely watertight argument.

Consider a traditional FM/AM radio. You can modify it, damage it, and get notable changes to its behaviour...

> Of note is anesthesia which can reliably and reversibly stop internal experience in the mind

...turn it off and on again...

> For a non-physical soul to hold our mind we would expect significant divergence from the above.

... yet concluding that all the noises produced from the radio are purely internal, mechanical and physical would be the wrong conclusion.

(I'm not arguing that the human brain/mind is anything like analogous to a radio, just pointing out the limits of this approach.)

Re: Language models can explain neurons in language models

#322

Post by p1esk »

karpierz wrote: post #320
p1esk wrote: post #313

Earlier quoted context omitted.

Interesting - do you believe average humans (not professional lawyers) would do better on this task?

Probably not. I'm not sure that pointing out that LLMs are as useful for parsing legal code as the average human is something to brag about though.

You're not sure that having an AI which is (at least) as intelligent as an average human is something to brag about? Seriously?

Re: Language models can explain neurons in language models

#323

Post by akiselev »

p1esk wrote: post #313
akiselev wrote: post #283

Earlier quoted context omitted.

Use it to analyze the California & US Code, the California & Federal Codes of Regulation, and bills currently in the California legislation & Congress. It's far from useless but far more useful for creative writing than any kind of understanding or instruction following when it comes to complex topics. Even performing a map-reduce over large documents to summarize or analyze them for a specific audience is largely be…

Interesting - do you believe average humans (not professional lawyers) would do better on this task?

Yes. I can parse them just fine after reading a single book called Introduction to Legal Reasoning [1]. I can also autonomously take notes and keep track of a large context using a combination of short and long term memory despite not having any kind of degree let alone experience or a license to practice law.

How do you think people become lawyers and how smart do you think the average lawyer actually is? The problem is that there's hundreds of thousands if not millions of pages, not that it requires superhuman intelligence to understand.

Even if it were capable of intelligence in the bottom quartile of humanity it would be SO MUCH more useful than it is now because I'd be able run and get something useful out of thousands of models in parallel. As it stands now GPT4 fails miserably at scaling up the kind of reasoning and understanding that even relatively stupid humans are capable of.

[1] https://www.amazon.com/Introduction-Legal-Reasoning-Edward-L...

Re: Language models can explain neurons in language models

#324

Post by DaiPlusPlus »

0xParlay wrote: post #145
TheRealPomax wrote: post #63

Earlier quoted context omitted.

Why is this genius? It's just the NN equivalent of making a new programming language and getting it to the point where its compiler can be written in itself. The reliability question is of course the main issue. If you don't know how the system works, you can't assign a trust value to anything it comes up with, even if it seems like what it comes up with makes sense.

I love the epistemology related discussions AI inevitably surfaces. How can we know anything that isn't empirically evident and all that. It seems NN output could be trusted in scenarios where a test exists. For example: "ChatGPT design a house using [APP] and make sure the compiled plans comply with structural/electrical/design/etc codes for area [X]". But how is any information that isn't testable trusted? I'm open…

> But how is any information that isn't testable trusted? I'm open to the idea ChatGPT is as credible as experts in the dismal sciences given that information cannot be proven or falsified and legitimacy is assigned by stringing together words that "makes sense".

I understand that around the 1980s-ish, the dream was that people could express knowledge in something like Prolog, including the test-case, which can then be deterministically evaluated. This does really work, but surprisingly many things cannot be represented in terms of “facts” which really limits its applicability.

I didn’t opt for Prolog electives in school (I did Haskell instead) so I honestly don’t know why so many “things” are unrepresentable as “facts”.

Re: Language models can explain neurons in language models

#325

Post by chimpanzee »

p1esk wrote: post #316
chimpanzee wrote: post #312

Earlier quoted context omitted.

Ah ok. Here you use the word “explain” which implies more of a descriptive, reducing action rather than extrapolative and constructive. As in, it can explain what it has “read” (and it has obviously “read” far more than any human), but it can’t necessarily extrapolate beyond that or use that to find new truths. To me reasoning is more about the extrapolative, truth-finding process, ie “wisdom” from knowledge rather t…

Why don't you suggest an example we can run and see what it's capable of (compared to what I, or other humans, are capable of)?

(In case it was missed, I’ve added a relevant addendum to my previous comment.)

Not sure an example is needed because I agree it “explains” better than pretty much everyone. (From my mostly lay perspective) It essentially uses the prompt as an argument in a probabilistic analysis of its incredibly vast store of prior inputs to transform them into an output that at least superficially satisfies the prompter’s goals. This is cool and useful, to say the least. But this is only one kind of reasoning.

A machine without embodied perceptual experiences simply cannot reason to the full-extent of a human.

(It’s also worth remembering that the prompter (very likely) has far less knowledge of the domain of interest and far less skill with the language of communication, so the prompter is generally quite easily impressed regardless of the truth of the output. Nothing wrong with that necessarily, especially if it is usually accurate. But again, worth remembering.)

Re: Language models can explain neurons in language models

#326

Post by lennxa »

davesque wrote: post #144

Based on my skimming the paper, am I correct in understanding that they came up with an elaborate collection of prompts that embed the text generated by GPT-2 as well as a representation of GPT-2's internal state? Then, in effect, they simply asked GPT-4, "What do you think about all this?" If so, they're acting on a gigantic assumption that GPT-4 actually correctly encodes a reasonable model of the body of knowledge…

Why does it have to understand how the LLMs are built? They have used gpt-4 to just build a classifier for each neuron's activition, and given the nlp abilities of gpt-4, the hope is that it can describe the nature of activation of the neurons.

Re: Language models can explain neurons in language models

#327

Post by robwwilliams »

derefr wrote: post #264
akiselev wrote: post #244

Earlier quoted context omitted.

If you spent even more time with GPT-4 it would be evident that it is definitely not. Especially if you try to use it as some kind of autonomous agent.

AI research has put hardly any effort into building goal-directed agents / A-Life since the advent of Machine Learning. A-Life was last really "looked into" in the '70s, back when "AI" meant Expert Systems and Behavior Trees. All the effort in AI research since the advent of Machine Learning, has been focused on making systems that — in neurological terms — are given a sensory stimulus of a question, and then passive…

Brilliant comment—-and back to basics.

Yes, and put that compact fruit fly in silico brain into my Roomba please so that it does not get stuck under the bed.

This is the kind of embodied AI that should really worry us.

Don’t we all suspect deep skunkworks “defense” projects of these types?

Re: Language models can explain neurons in language models

#328

Post by karpierz »

p1esk wrote: post #322
karpierz wrote: post #320

Earlier quoted context omitted.

Probably not. I'm not sure that pointing out that LLMs are as useful for parsing legal code as the average human is something to brag about though.

You're not sure that having an AI which is (at least) as intelligent as an average human is something to brag about? Seriously?

Where did you get that the LLM was as _intelligent_ as a human?

All we've shown is that LLMs are as useful for parsing legal text as the average human. Which is to say, not.

A dog is also as useful for parsing legal texts as the average human. So is a rock.

Re: Language models can explain neurons in language models

#329

Post by galaxyLogic »

incangold wrote: post #96
galaxyLogic wrote: post #54

Earlier quoted context omitted.

It would if the language model did reasoning according rules of logic. But they don't. They use Markov chains. To me it makes no sense to say that a LLM could explain its own reasoning if it does no (logical) reasoning at all. It might be able to explain how the neural network calculates its results. But there are no logical reasoning steps in there that could be explained, are there?

Honest question: are we sure that it doesn’t do logical reasoning? IANAE but although an LLM meets the definition of a Markov Chain as I understand it (current state in, probabilities of next states out), the big black box that spits out the probabilities could be doing anything. Is it fundamentally impossible for reasoning to be an emergent property of an LLM, in a similar way to a brain? They can certainly do a goo…

> Honest question: are we sure that it doesn’t do logical reasoning?

It's not the Creature from the Lagoon, its an engineering artifact created by engineers. I haven't heard them say it does logical deduction according to any set of logic-rules. What I've read is it uses Markov chains. That makes sense because basically an LLM given a string-input should reply with another string that is the most likely follow-up string to the first string, based on all the texts it crawled up from the internet.

If internet had lots and lots of logical reasoning statements then a LLM might be good at producing what looks like logical reasoning, but that would still be just response with the most likely follow-up string.

The reason the results of LLMs are so impressive is that at some point the quantity of the data makes a seemingly qualitative difference. It's like if you have 3 images and show them each to me one after the other I will say I saw 3 images. But if you show me thousands of images 24 per second and the images are small variations of the previous images then I say I see a MOVING PICTURE. At some point quantity becomes quality.

Re: Language models can explain neurons in language models

#330

Post by streakfix »

akiselev wrote: post #323
p1esk wrote: post #313

Earlier quoted context omitted.

Interesting - do you believe average humans (not professional lawyers) would do better on this task?

Yes. I can parse them just fine after reading a single book called Introduction to Legal Reasoning [1]. I can also autonomously take notes and keep track of a large context using a combination of short and long term memory despite not having any kind of degree let alone experience or a license to practice law. How do you think people become lawyers and how smart do you think the average lawyer actually is? The proble…

Did you try fine tuning gpt4 with that book as input?
Post reply on HN