Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Language models can explain neurons in language models

openai.com

Post reply on HN
Print view View original

451–460 of 497 posts

Re: Language models can explain neurons in language models

#451

Post by haldujai »

ethanbond wrote: post #442
haldujai wrote: post #427

Earlier quoted context omitted.

You're ignoring several confounders and conflating several loosely related things that I'm not even sure what the point you're making is anymore. To begin with, the split-brain experiments don't provide clear or strong evidence for anything given the small sample size, heterogeneity in procedure (i.e. was there complete comissurotomy or just callosotomy) and the elapsed time between neuropsychiatric evaluation and in…

I'm not "ignoring" them at all. I'm saying that they point to interesting questions that are not answered. > The split-brain experiments are notable because the lab experiments SUGGEST the lack of communication between two hemispheres and a split conscious however this is paradoxical with everyday experience of these patients, far from providing evidence for anything. It is not "paradoxical" but yes it does conflict…

I'm arguing against the strength of your statements based on methodologically unsound experiments that do not "very very clearly indicate" anything beyond pose a few questions for which there are several different hypothetical answers. All of which have zero evidence behind them.

Similarly, the initial comment of 'does the brain know what the brain is doing. The answer so far does not seem to be "yes."' is misleadingly suggesting there is a shred of evidence supporting that the answer is 'no' or that the answer is 'not yes'. There are no answers so far, just questions.

If anything, there are more unified consciousness hypotheses than otherwise, although if you refer back to my original reply I did not make this assertion: 'I think the correct statement is "so far the answer is we don't know"'

> It is not "paradoxical" but yes it does conflict with some reported experience.

Rather than belabour the experiment results and implications here is a great peer-reviewed article by experts in the field: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7305066/

> Regarding addiction, it is very, very commonly reported that addicts will go into "autopilot" like states while satisfying their addictions and only "emerge" when they have to face consequences of their behaviors. Again, subjectively reported, but so is the experience of unitary consciousness! If we cannot trust one then we shouldn't take it as granted that we can trust the other.

The dopamine reward system understanding (which by the way is probably the most well-validated and widely believed model in neurobiology) provides a rational explanation for addiction.

You haven't explained what is self-contradictory, that a few case reports exist of patients claiming they went in and out of consciousness? That's not a contradiction.

Re: Language models can explain neurons in language models

#452

Post by hanspeter »

jwestbury wrote: post #414
mr_toad wrote: post #236

Earlier quoted context omitted.

> There is no evidence that intelligence runs on neurons. 1. Neurons connect all our senses and all our muscles. 2. Neurons are the definitive difference between the brain and the rest of the body. There is “other stuff” in the brain, but it’s not so different from the “other stuff” that’s in your rear end. Don’t underestimate what a neuron can do. A single artificial neuron can fit a logistic regression model. A qua…

> 2. Neurons are the definitive difference between the brain and the rest of the body. There is “other stuff” in the brain, but it’s not so different from the “other stuff” that’s in your rear end. Our digestive systems appear to be important to our behaviour, though. Some recent work in mice showed that if colonised with bacteria from faeces of humans with autism, the mice would begin to show autistic behaviours. So…

The role that digestive systems potentially play for our behavior is via neurotransmitters, so in the end it is the neurons, that are responsible.

If the study you mention show something else, I would actually be very interested in a source.

Re: Language models can explain neurons in language models

#453

Post by marshray »

jmfldn wrote: post #157
marshray wrote: post #147

Earlier quoted context omitted.

> There is no self reflection, but if you ask an LLM program how "it" knows something it will produce some text. To be clear, you're saying that we should just dismiss out-of-hand any possibility that an LM AI might actually be able to explain its reasoning step-by-step? I find it kind of charming actually how so many humans are just so darn sure that they have their own special kind of cognition that could never be…

That's a strawman since I didn't argue anything about humans being special. I don't think there is anything necessarily inherently special about human intelligence, I'm just advocating for caution around the language we use to talk about current systems. All this talk of AGI and sentience and so on is premature and totally unfounded . It's pure sci fi, for now at least.

> I didn't argue anything about humans being special

Above you said about AI LMs:

> There is no "their" and there is no "thought process"

So, unless you're claiming that humans lack a thought process as well, then you're arguing that humans are special.

> All this talk of AGI and sentience and so on is premature and totally unfounded

I don't see any mention of AGI or sentience in this thread?

Also, I don't think anyone could read this transcript with GPT-4 and still claim that it's incapable of a significant degree of self-reflection and metacognition:

https://news.ycombinator.com/item?id=35880148

Re: Language models can explain neurons in language models

#454

Post by YeGoblynQueenne »

p1esk wrote: post #219
techolic wrote: post #215

Earlier quoted context omitted.

> that we're pretty damn close to recreating it Is that evident already or are we fitting the definition of intelligence without being aware?

If you spent any time with GPT-4 it should be evident.

I watched a magician's show where he made his assistant hover in mid-air. It is evident that the magician has mastered levitation.

Re: Language models can explain neurons in language models

#455

Post by nerpderp82 »

YawningAngel wrote: post #76
nerpderp82 wrote: post #64

Earlier quoted context omitted.

What if you ask it to emit the reflexive output, then feed that reflexive output back into the LLM for the conscious answer? What if you ask it to synthesize multiple internal streams of thought, for an ensemble of interior monologues, then have all those argue with each other using logic and then present a high level answer from that panoply of answers?

What if you do? LLMs don't have reflexive output or internal streams of thought, they are simply (complex) processes that produce streams of tokens based on an inputted stream of tokens. They don't have a special response to tokens that indicate higher-level thinking to humans.

You seem to have high confidence in how LLMs work.

Re: Language models can explain neurons in language models

#456

Post by YeGoblynQueenne »

int_19h wrote: post #447
runlaszlorun wrote: post #418

Earlier quoted context omitted.

But it’s also based on neurons with far more complex behavior than artificial neurons and also has other separate dynamic systems involving neurochemicals, various effects across the nervous system and the rest of the body (the gut becoming seemingly more and more relevant), various EEG patterns, and most likely quantum effects. I personally wouldn’t rule out that it can’t be emulated in a different substrate, but I…

If it performs a computation, it is by definition running some algorithm regardless of how it's implemented in hardware / wetware. How is it a stretch? The only way our brains could be not algorithmic is if something like soul is a real thing that actually drives our intelligence.

Why? Rain is not algorithmic, clouds are not algorithmic, waves in the sea are not algorithmic, yet they are entirely physical processes that have nothing to do with souls.

Re: Language models can explain neurons in language models

#457

Post by AdieuToLogic »

int_19h wrote: post #447
runlaszlorun wrote: post #418

Earlier quoted context omitted.

But it’s also based on neurons with far more complex behavior than artificial neurons and also has other separate dynamic systems involving neurochemicals, various effects across the nervous system and the rest of the body (the gut becoming seemingly more and more relevant), various EEG patterns, and most likely quantum effects. I personally wouldn’t rule out that it can’t be emulated in a different substrate, but I…

If it performs a computation, it is by definition running some algorithm regardless of how it's implemented in hardware / wetware. How is it a stretch? The only way our brains could be not algorithmic is if something like soul is a real thing that actually drives our intelligence.

> The only way our brains could be not algorithmic is if something like soul is a real thing that actually drives our intelligence.

Therein lies the question, one which deserves contemplation and can lead to Enlightenment.

Which then begs the question; is Enlightenment a "real thing" and, if not, how is it that it can be experienced?

Re: Language models can explain neurons in language models

#458

Post by YeGoblynQueenne »

p1esk wrote: post #319
chimprich wrote: post #309

Earlier quoted context omitted.

Google search is better than reasoning than most humans - in that if you search for an explanation of something then Google's first result is often correct, or one of the following ones. GPT-4 will often come up with a solution to a problem, but only if it has learnt something similar (it's better than Google in some respects: it can extract and combine abstractions). However, both need handholding by a human (supply…

OK, you have a point. But, if you replace "GPT-4" with "an average human" in your response - it still makes sense.

When you ask a question of "an average human", they will reply with a sentence answering your question, or maybe asking for clarifications, etc. When you ask a question of GPT-4, it generates one token.

One. Token.

One token, that is not an answer to your question, that is not asking for clarifications, that is not trying to reason about your question, or how to answer it, because it's just one token.

One. Token.

For example, say you ask:

  "What is the sieve of Eratosthenes? Speak in chicken".
Then GPT-4 will answer with something like:

  Chicken
You don't see that because you don't interact with the GPT-4 model directly. You interact with ChatGPT, a "wrapper" around GPT-4, that takes your prompt, passes it to the model, then takes the token returned by the model and decides what to do with it. In the example above, ChatGPT will take the " Chicken" token generated by GPT-4 and append it to your question:

  What is the sieve of Eratosthenes? Speak in chicken. Chicken
Then it will send this new string back to the model, which will generate a new token:

  chicken
And then it will go like this:

  Iteration 2: What is the sieve of Eratosthenes? Speak in chicken. Chicken chicken

  ... 

  Iteration k: What is the sieve of Eratosthenes? Speak in chicken. Chicken chicken chicken chicken chicken chicken chicken chicken chicken chicken ...
At no point is GPT-4 trying to reason about your question, or try to answer your question, or do anything else than generate one. token. at a time. There's no thinking, no reasoning, no calculation, no logic, no deduction, no intelligence, no anything. It's only token, token, token. Chicken, chicken, chicken.

And do you know when the chickens stop? When GPT-4 generates a special and magickal token, called a stop-token (or a "stop sequence" in OpenAI docs). That's a token, not found in the training corpus, added to the end of every string during tokenisation. That's how ChatGPT knows to stop sending back your prompt + generated tokens, to the model. It can't look back to what GPT-4 has generated so far, because it doesn't understand any of that. Because it doesn't understand anything, and therefore cannot reason about your question, or realise it has answered it. It cannot do anything except a) ask GPT-4 to generate another token or b) stop asking for more tokens.

  "What is the sieve of Eratosthenes? Speak in chicken". Chicken, chicken, chicken, chicken, .... chicken, stop_token!
No more chickens.

And that's how GPT-4 explains what the Seive of Eratosthenes is, but in chicken.

So what you see as a user is like watching a movie where a plate of spaghetti is flying through the air, lands on a table cloth, the tablecloth lands on a table, the table lands on the floor, four chairs land around it and suddendly there's people in the chairs eating the spaghetti. It's not that someone has figured out how to reverse time: it's a recording, played backwards. It looks like things are going backwards, but they're not.

It looks like ChatGPT is answering your questions, but it's not. It looks like ChatGPT is interacting with you, but it's not. It's a magick trick.

Re: Language models can explain neurons in language models

#460

Post by jxmorris12 »

MacsHeadroom wrote: post #216
davesque wrote: post #144

Based on my skimming the paper, am I correct in understanding that they came up with an elaborate collection of prompts that embed the text generated by GPT-2 as well as a representation of GPT-2's internal state? Then, in effect, they simply asked GPT-4, "What do you think about all this?" If so, they're acting on a gigantic assumption that GPT-4 actually correctly encodes a reasonable model of the body of knowledge…

>If so, they're acting on a gigantic assumption Yes the initial hypothesis that GPT-4 would know was a gigantic assumption. But a falsifiable one which we can easily generate reproducible tests for. The idea that simulated neurons could learn anything useful at all was once a gigantic assumption too.

Neural networks aren’t simulated neurons…
Post reply on HN