Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Language models can explain neurons in language models

openai.com

Post reply on HN
Print view View original

371–380 of 497 posts

Re: Language models can explain neurons in language models

#371

Post by chrisco255 »

creatonez wrote: post #364
chrisco255 wrote: post #137

Earlier quoted context omitted.

No, it wasn't, language itself didn't even exist at one point. It wasn't inferred from training data into existence because such examples existed before. Now we have a dictionary of tens of thousands of words, which describe high level ideas, abstractions, and concepts that someone, somewhere along the line had to invent. And I'm not talking about imitation nor am I interested in semantic games, I'm talking about raw…

Coming up with new language is rarely ever coming up with new concepts that didn't exist until the word. We come up with high-level abstractions because there already exists a material system to be described and modeled. Language that doesn't describe anything that already exists is more like babbling.

Not really, no. There are plenty of intangible abstractions that don't describe material systems. Take imaginary numbers, for example. Or the concept of infinity or even zero, neither of which exists in the physical world.

The reason why "naming things" is the other hard problem in computer science, after cache invalidation, is that the process of identifying, creating, and describing ideal abstractions is itself inherently difficult.

Re: Language models can explain neurons in language models

#372

Post by chimpanzee »

p1esk wrote: post #342
chimpanzee wrote: post #325

Earlier quoted context omitted.

(In case it was missed, I’ve added a relevant addendum to my previous comment.) Not sure an example is needed because I agree it “explains” better than pretty much everyone. (From my mostly lay perspective) It essentially uses the prompt as an argument in a probabilistic analysis of its incredibly vast store of prior inputs to transform them into an output that at least superficially satisfies the prompter’s goals. T…

What would be an example of “non-deductive” reasoning, which requires embodied perceptual experiences?

“God, that felt great!”

As detailed as possible, describe what happened.

Re: Language models can explain neurons in language models

#373

Post by KineticLensman »

_justinfunk wrote: post #245
candiddevmike wrote: post #234

Earlier quoted context omitted.

Because we eat and breath through the same tube

There are two tubes.

The trachea and oesophagus both start in the common space of the pharynx. The epiglottis stops food going down the wrong one

Re: Language models can explain neurons in language models

#375

Post by sjy »

streakfix wrote: post #330
akiselev wrote: post #323

Earlier quoted context omitted.

Yes. I can parse them just fine after reading a single book called Introduction to Legal Reasoning [1]. I can also autonomously take notes and keep track of a large context using a combination of short and long term memory despite not having any kind of degree let alone experience or a license to practice law. How do you think people become lawyers and how smart do you think the average lawyer actually is? The proble…

Did you try fine tuning gpt4 with that book as input?

Fine-tuning requires you to train the model with a set of prompts and desired completions. Building a suitable dataset is not trivial and it's not clear what it would mean to use a book for fine-tuning anyway – masking sentences and paragraphs and training the model to complete them in the book's style?

Re: Language models can explain neurons in language models

#376

Post by dpflan »

PeterisP wrote: post #117
jablongo wrote: post #56

Earlier quoted context omitted.

Right so it should be much easier w/ access to every neuron and activation. But the general approach is an experimental one where you try to use your existing knowledge about physics and biology to discern what is activating different structures (and neurons) in the brain. I agree w/ the approach of trying to assign some functionality to individual 'neurons', but I don't think that using GPT4 to do so is the most app…

All of this seems to lead to something like this paper https://journals.plos.org/ploscompbiol/article?id=10.1371/jo... On the other hand, I find it plausible that it's fundamentally impossible to assign some functionality to individual 'neurons' due to the following argument: 1. Let's assume that for a system calculating a specific function, there is a NN configuration (weights) so that at some fully connected NN lay…

Your last idea, that these models’ neurons are all connected in some way, makes me somewhat sceptical of this research by OpenAI. And that their technique of analysis may need to be more fractal or expansive to include groups of neurons, moving all the way up to the entire model.

Re: Language models can explain neurons in language models

#377

Post by ly3xqhl8g9 »

Aerbil313 wrote: post #360
stevenhuang wrote: post #253

Earlier quoted context omitted.

Perhaps there is no purpose to consciousness. Perhaps it's a phenomenon that somehow arises independently ex nihilo from sufficiently complex systems, only ever able to observe, unable to act. Weird to think about.

Can conscious experience ever arise from matter? Even if the said matter is neural networks? This seems utterly nonsensical to me.

We are composed of matter and we are conscious, we take this as being axiomatic. Given we have one certain example, the question is then how does the body do it, and how could we do it in other substrates: how do we go from "just physics" to mind [1].

The short answer is: chemical reactions start a chain reaction of abstraction towards higher and higher forms of collective intelligence.

For some reason, perhaps something with the way the Hilbert space vector obeying the Schrödinger equation which we usually call "the universe" is [2], but also given the ridiculous breadth and depth of possible explorations of the biological pathways [3], "chunks" of matter tend to group together, they group and form stars and planets, but they also group and form formaldehydes and acetaldehydes and many more. Given enough tries, across multiple environments, in some lucky hydrothermal vents abiogenesis was probably started [4]. Once we had the first agential "chunk" of matter, a group of matter which has a definition of the boundary between a "self", no matter how tiny [5], and an exterior environment, it was more of a game of waiting (~4 billion years) for this tiny agent to grow into an agent with higher-order thinking, self-referentiality, metacognition, and the likes.

Neural networks, as in matrix multiplications, are not conscious because they have no mechanism for deciding what is the environment and what is their own, they are a hammer, sitting there, expecting to be used, not a lacrymaria olor [6], exploring the environment for survival and fun. Could we have neural networks in an agent-like architecture starting to behave more like thermostats, setting goals for themselves? Probably.

[1] "From physics to mind - Prof. Michael Levin", https://youtu.be/_QICRPFWDpg?t=85

[2] "Sean Carroll: Extracting the universe from the wave function", https://www.youtube.com/watch?v=HOssfva2IBo

[3] "Nick Lane, 'Transformer : The Deep Chemistry of Life and Death'", https://youtu.be/bEFzUx_j7tA?t=279

[4] https://en.wikipedia.org/wiki/Hydrothermal_vent

[5] https://en.wikipedia.org/wiki/Physarum_polycephalum

[6] https://www.youtube.com/watch?v=sq6Y54mxjOg

Re: Language models can explain neurons in language models

#378

Post by chaxor »

poulpy123 wrote: post #362
p1esk wrote: post #357

Earlier quoted context omitted.

Do you have any example of how it cannot reason?

It's a neural network. Neural network are not symbolic AI and are not designed to reason

There's a decent working paper that has benchmarks on this, if you're interested.

There are many types of reasoning, but GPT-4 gets 97% on casual discovery, and 92% on counterfactuals (only 6% off from human, btw) with 86% on actual causality benchmarks.

I'm not sure yet if the question is correct, or even appropriate/achievable to what many may want to ask (i.e. what 'the public's is interested in is typically lost after it is defined in any given study); however this is one of the best works available to address this problem I've seen so far, so perhaps it can help.

Re: Language models can explain neurons in language models

#379

Post by DaiPlusPlus »

philomath_mn wrote: post #344
DaiPlusPlus wrote: post #324

Earlier quoted context omitted.

> But how is any information that isn't testable trusted? I'm open to the idea ChatGPT is as credible as experts in the dismal sciences given that information cannot be proven or falsified and legitimacy is assigned by stringing together words that "makes sense". I understand that around the 1980s-ish, the dream was that people could express knowledge in something like Prolog, including the test-case, which can then…

I bet GPT is really good at prolog, that would be interesting to explore. "Answer this question in the form of a testable prolog program"

Did you give it a try?

Re: Language models can explain neurons in language models

#380

Post by posterboy »

mxschll wrote: post #349

> We are open-sourcing our datasets and visualization tools for GPT-4-written explanations of all 307,200 neurons in GPT-2, as well as code for explanation and scoring using publicly available models on the OpenAI API. We hope the research community will develop new techniques for generating higher-scoring explanations and better tools for exploring GPT-2 using explanations. Aww, that's so nice of them to let the com…

Surely they are willing to shell a out a few bucks
Post reply on HN