Live data from Hacker News

Language models can explain neurons in language models

openai.com

381–390 of 497 posts

Re: Language models can explain neurons in language models

#381
post #357

Earlier quoted context omitted.

Do you have any example of how it cannot reason?

It's a neural network. Neural network are not symbolic AI and are not designed to reason

It doesn't matter what it is designed for. What matters is what it actually does. It's trivial to get GPT to do chain-of-thought reasoning and observe it.

Re: Language models can explain neurons in language models

#382
post #315

Earlier quoted context omitted.

I am not. I treat it as I'd treat any smart human being.

LLM's are not a "smart human being." They are predictive statistical models capable of producing results based on training data. LLM's do not think. LLM's are algorithms.

Your brain is also basically an algorithm that produces results based on training data. It's just a much more complicated and flexible one.

Re: Language models can explain neurons in language models

#383

Earlier quoted context omitted.

Basically, yeah.

Notable that the people who saw the first automatons argued whether they were alive.

People also argued that animals are mere automatons, that all their observable behavior is basically scripted, and that humans shouldn't empathize with them on those grounds. That's how vivisection etc used to be justified since at least Descartes.

Re: Language models can explain neurons in language models

#384

Earlier quoted context omitted.

People are better understood intuitively. We understand how people fail and why. We can build trust with people with some degree of success. But machine models are new and can fail in unpredictable ways. They also get deployed to billions of users in a way that humans do not, and deployed in applications that humans do not. So its certainly useful to try to explain neural networks in as great of detail as we can.

Or we can build trust using black box methods like we do with humans, e.g., extrapolating from past behavior, administering tests, and the like.

We can, but the nice thing about neural networks is the ability to do all kinds of computational and mathematical manipulations to them to basically pick them apart and really find out what’s going on. This is important not just for safe deployment but also for research on new methods that could be used to make them better. Plus we need this ability to help avoid neural networks with intentionally hidden features that appear to behave linearly in certain regimes but are designed with a strong nonlinear response when special inputs are applied. You could have all the tests you want for a self driving car based on real world conditions but some bad actor with access to the training system could create a special input that results in dangerous behavior.

Re: Language models can explain neurons in language models

#385

Earlier quoted context omitted.

Brain damage by physical trauma, disease, oxygen deprivation, etc. has dramatic and often permanent effects on the mind. The effect of drugs (including alcohol) on the mind. Of note is anesthesia which can reliably and reversibly stop internal experience in the mind. For a non-physical soul to hold our mind we would expect significant divergence from the above. Out of body experiences and similar are indistinguishabl…

> Brain damage by physical trauma, disease, oxygen deprivation, etc. has dramatic and often permanent effects on the mind. That's not a completely watertight argument. Consider a traditional FM/AM radio. You can modify it, damage it, and get notable changes to its behaviour... > Of note is anesthesia which can reliably and reversibly stop internal experience in the mind ...turn it off and on again... > For a non-phys…

I mean, if we're really going to go there, who's to say that a large enough LLM doesn't automatically receive a soul simply because that's one of the fundamental laws of the universe as decreed by the Creator?

Re: Language models can explain neurons in language models

#386

Earlier quoted context omitted.

If you really want to present ants as a civilization, I don't think a single ant is a meaningful unit of that civilization comparable to a single human. A colony, perhaps - but then that's a lot more neurons, just distributed.

A single human will die permanently, alone, miserable, and unproductive. The comparison is apt.

A single human can survive on their own; there are many historical examples of that. A detached body part, on the other hand, cannot; but it also cannot feel miserable etc. A single ant is more like a body part of the colony in that sense.

Re: Language models can explain neurons in language models

#387

Earlier quoted context omitted.

Or we can build trust using black box methods like we do with humans, e.g., extrapolating from past behavior, administering tests, and the like.

We can, but the nice thing about neural networks is the ability to do all kinds of computational and mathematical manipulations to them to basically pick them apart and really find out what’s going on. This is important not just for safe deployment but also for research on new methods that could be used to make them better. Plus we need this ability to help avoid neural networks with intentionally hidden features tha…

The more fundamental problem is the sheer size of them, and this is only going to get worse as models grow larger to become more capable. Being able to look at the state of individual neurons during inference is very convenient, but that does not by itself make it possible to really find out what's going on.

Re: Language models can explain neurons in language models

#388
post #375

Earlier quoted context omitted.

Did you try fine tuning gpt4 with that book as input?

Fine-tuning requires you to train the model with a set of prompts and desired completions. Building a suitable dataset is not trivial and it's not clear what it would mean to use a book for fine-tuning anyway – masking sentences and paragraphs and training the model to complete them in the book's style?

> masking sentences and paragraphs and training the model to complete them in the book's style?

That would work.

Re: Language models can explain neurons in language models

#390

Earlier quoted context omitted.

> Brain damage by physical trauma, disease, oxygen deprivation, etc. has dramatic and often permanent effects on the mind. That's not a completely watertight argument. Consider a traditional FM/AM radio. You can modify it, damage it, and get notable changes to its behaviour... > Of note is anesthesia which can reliably and reversibly stop internal experience in the mind ...turn it off and on again... > For a non-phys…

I mean, if we're really going to go there, who's to say that a large enough LLM doesn't automatically receive a soul simply because that's one of the fundamental laws of the universe as decreed by the Creator?

Going where? I wasn't arguing for the existence of a soul.

Although, sure, if we could somehow manage to determine that souls did exist then presumably an AI model as capable as a human would also be eligible for one.

Post reply on HN