Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Language models can explain neurons in language models

openai.com

Post reply on HN
Print view View original

361–370 of 497 posts

Re: Language models can explain neurons in language models

#361

Post by FeepingCreature »

lubesGordi wrote: post #126
ethanbond wrote: post #121

Earlier quoted context omitted.

No it’s a given that some people who attempt to wield AI will be adversarial. In any case a similar argument can be made with merely instrumental goals causing harm: “I am an ant and I do not see how or why a human would cause me harm, therefore I am not in danger.”

People wielding AI and destroying humanity is very different from AI itself, being a weird alien intelligence, destroying humanity. Honestly if you have no examples you can't really blame people for not being scared. I have no reason to think this ant-human relationship is analogous. And seriously, I've made no claims that AI is benign so please stop characterizing my claims thusly. The question is simple, give me a…

If AI is general, it will also have the capability of wielding itself, and probably better than humans.

Re: Language models can explain neurons in language models

#362

Post by poulpy123 »

p1esk wrote: post #357
poulpy123 wrote: post #352

Earlier quoted context omitted.

While impressive GPT-4 isn't intelligent or close to intelligence. It's an impressive statistical model but doesn't have the ability to reason

Do you have any example of how it cannot reason?

It's a neural network. Neural network are not symbolic AI and are not designed to reason

Re: Language models can explain neurons in language models

#363

Post by FeepingCreature »

tschumacher wrote: post #199

Even if we can explain the function of a single neuron what do we gain? If the goal is to reason about safety of computer vision in automated driving as an example, we would need to understand the system as a whole. The whole point of neural networks is to solve nuanced problems we can't clearly define. The fuzziness of the problems those systems solve is fundamentally at odds with the intent to reason about them.

If there is a limited number of deception neurons, we can automatically look for coactivation of the "my future behavior" neuron and the deception neuron.

Re: Language models can explain neurons in language models

#364

Post by creatonez »

chrisco255 wrote: post #137
PeterisP wrote: post #106

Earlier quoted context omitted.

Tautologically, every concept that anything (LLM, or human, or alien) develops can be inferred from the input data(e.g. training set), because it was.

No, it wasn't, language itself didn't even exist at one point. It wasn't inferred from training data into existence because such examples existed before. Now we have a dictionary of tens of thousands of words, which describe high level ideas, abstractions, and concepts that someone, somewhere along the line had to invent. And I'm not talking about imitation nor am I interested in semantic games, I'm talking about raw…

Coming up with new language is rarely ever coming up with new concepts that didn't exist until the word. We come up with high-level abstractions because there already exists a material system to be described and modeled. Language that doesn't describe anything that already exists is more like babbling.

Re: Language models can explain neurons in language models

#365

Post by [deleted] »

p1esk wrote: post #342
chimpanzee wrote: post #325

Earlier quoted context omitted.

(In case it was missed, I’ve added a relevant addendum to my previous comment.) Not sure an example is needed because I agree it “explains” better than pretty much everyone. (From my mostly lay perspective) It essentially uses the prompt as an argument in a probabilistic analysis of its incredibly vast store of prior inputs to transform them into an output that at least superficially satisfies the prompter’s goals. T…

What would be an example of “non-deductive” reasoning, which requires embodied perceptual experiences?

[deleted]

Re: Language models can explain neurons in language models

#366

Post by chrisco255 »

flangola7 wrote: post #164
chrisco255 wrote: post #137

Earlier quoted context omitted.

No, it wasn't, language itself didn't even exist at one point. It wasn't inferred from training data into existence because such examples existed before. Now we have a dictionary of tens of thousands of words, which describe high level ideas, abstractions, and concepts that someone, somewhere along the line had to invent. And I'm not talking about imitation nor am I interested in semantic games, I'm talking about raw…

> No, it wasn't, language itself didn't even exist at one point. Language took dozens of millennia to form, and animals have long had vocalizations. Seems like a natural building on top of existing features. > Has AI ever managed to learn something humans didn't already know? AlphaZero invented all new categories of strategy for games like Go, when previously we thought almost all possible tactics had been discovered…

> Language took dozens of millennia to form

Sure, but in a simulated evolutionary algorithm, you can hit a few hundred generations in a matter of seconds.

Indeed, the identification of an abstraction, followed by a definition of that abstraction and an enshrinement of the concept in the form of a word or phrase, in and of itself, shortcuts the evolutionary path altogether. AI isn't starting from scratch: it's starting from a dictionary larger than any human alive knows and in-memory examples of humans conversing on nearly every topic imaginable.

We never thought "all possible tactics" had been discovered with Go. We quite literally understood that Go had a more complex search space than Chess, with far more possible moves and outcomes. And I don't think anyone has any kind of serious theorem that "all possible tactics" have been discovered in either game, to this day.

That being said, Go and Chess are enumerable games with deterministic, bounded complexity and state space.

The protein folding example is a neat one, I definitely think it's interesting to see what develops there. However, protein folding has been modeled by Markov State models for decades. The AlphaFold breakthrough is fantastic, but it was already known how to generate models from protein structures: it was just computationally expensive.

It was also carefully crafted by humans to achieve what it did: https://www.youtube.com/watch?v=gg7WjuFs8F4. So this is an example of humans using neural network technology that humans invented to achieve a desired solution to a known problem that they themselves conceived. The AI didn't tell us something we didn't already know. It was an expert system built with teams of researchers in the loop the whole way through.

Re: Language models can explain neurons in language models

#367

Post by chaxor »

coldtea wrote: post #161
int_19h wrote: post #139

Earlier quoted context omitted.

You don't even have to look that far ahead. Apparently, people are already using ChatGPT to compile custom diet plans for themselves, and they expect it to take into account the information they supply regarding their allergies etc. But, yes, those are also good examples of what we shouldn't be doing, but are going to do anyway.

Those cases sound like Darwin Awards mediated by high technology

I hear crushed glass has very great health benefits

Re: Language models can explain neurons in language models

#368

Post by callesgg »

creatonez wrote: post #355
callesgg wrote: post #120

Earlier quoted context omitted.

The text output of a llm is the thought process. In this context the main difference between humans and llms, is that llms can’t have internalized thoughts. There are of course other differences to, like the fact that humans have a wider gamut of input: visuals, sound, input from other bodily functions. And the fact that we have live training.

It's not clear whether or not LLMs have internal thoughts -- each token generation could absolutely have a ton of thought-like modelling in the hidden layers of the network. What is known is that these internal thoughts get erased each time a new token is generated. That is, it's starting from scratch from the contents of the text each time it generates a word. But you could postulate that similar prompt text leads t…

True, LLMs definetly has something that is "thought-like".

But todays networks lacks the recursion(feedback where the output can go directly to the input) that is needed for the type of internalized thoughts that humans have. I guess this is one thing you are pointing at by mentioning the continuousnes of the internals of LLMs.

Re: Language models can explain neurons in language models

#369

Post by creatonez »

ftxbro wrote: post #45

I wonder will someone please check the neurons associated to the petertodd and other anomalous glitch tokens ( https://www.lesswrong.com/posts/jkY6QdCfAXHJk3kea/the-petert... )? I can see the github and I see that for any given neuron you can see associated tokens but I don't see how to do an inverse search.

I think this method will do a poor job at explaining petertodd. These neuron explanations still have to fit within human language for this method to work, and the best you can do in the confines of human language to describe petertodd is to write a long article (just like that link) explaining the many oddities of it.

Would be interesting to try, though. I think it's likely that, due to the way glitch tokens happen, petertodd is probably an input neuron that is very randomly connected to a bunch of different hidden neurons. So it introduces some bizzare noise into a bunch of areas of the network. It's possible that some of these neurons are explainable on their own, but not within the broader context of petertodd.

Re: Language models can explain neurons in language models

#370

Post by chaxor »

gitfan86 wrote: post #221
talentedcoin wrote: post #193

Earlier quoted context omitted.

There is no evidence that any of those are emergent properties. It’s no more or less logical than asserting they were placed there by a creator.

There is evidence that people believe that GTP-4 is intelligent since it can solve things like the SATs. But if you start taking away weights one by one at some point those same people will say it isn't intelligent. A NN with 3 weights cannot solve any problems that humans believe requires intelligence. So where did it come from? I don't know, but it clearly emerged as the NN got bigger.

It's possible that the comment you're responding to is referring to the new paper out of Stanford/Google on emergence as a 'mirage'. The gist of it is that the shape of the curve for 'metric' vs 'params' is dependent upon the type of metric. For classification it has a certain shape, for accuracy it has another, etc. The authors claim the observation that these curves are determined by the choice of metric means that it isn't true emergence, but rather is due to the way in which GPTs are typically trained to predict the next token (and then beam search or typical sampling is often applied afterwards, etc).

The paper is somewhat new so I haven't done a proper review to know if it's solid work yet, but this may offer some context for some of the comments in this thread.

Post reply on HN