Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Language models can explain neurons in language models

openai.com

Post reply on HN
Print view View original

131–140 of 497 posts

Re: Language models can explain neurons in language models

#131

Post by cschmid »

simonw wrote: post #11
cschmid wrote: post #4

Has anyone here found a link to the actual paper? If I click on 'paper', I only see what seems to be an awkward HTML version.

You mean this? https://openaipublic.blob.core.windows.net/neuron-explainer/... Would you prefer a PDF? (I'm always fascinated to hear from people who would rather read a PDF than a web-native paper like this one, especially given that web papers are actually readable on mobile devices. Do you do all of your reading on a laptop?)

My whole workflow of organizing and reading papers is centered on PDFs. While I like having interactive supplemental materials, I want to be able to print, save and annotate the papers I read.

Re: Language models can explain neurons in language models

#132

Post by ethanbond »

lubesGordi wrote: post #126
ethanbond wrote: post #121

Earlier quoted context omitted.

No it’s a given that some people who attempt to wield AI will be adversarial. In any case a similar argument can be made with merely instrumental goals causing harm: “I am an ant and I do not see how or why a human would cause me harm, therefore I am not in danger.”

People wielding AI and destroying humanity is very different from AI itself, being a weird alien intelligence, destroying humanity. Honestly if you have no examples you can't really blame people for not being scared. I have no reason to think this ant-human relationship is analogous. And seriously, I've made no claims that AI is benign so please stop characterizing my claims thusly. The question is simple, give me a…

Sure, here’s a trivial example: It radicalizes or otherwise deceives an employee at a virus research lab into producing and releasing a horrific virus.

The guy at Google already demonstrated that AIs are able to convince people of fairly radical beliefs (and we have proof that even humans a thousand years ago were capable of creating belief systems that cause people to blow themselves up and kill thousands of innocent people).

P.S. I was not characterizing your opinion, I was speaking in the voice of an ant.

Re: Language models can explain neurons in language models

#134

Post by ethanbond »

kevinventullo wrote: post #123
ethanbond wrote: post #97

Earlier quoted context omitted.

This is also why I go into chess matches against 1400 elo players. I cannot conceive of the specific ways in which they will beat me (a 600 elo player), so I have good reason to suspect that I can win. I'm willing to bet the future of our species on my consistent victory in these types of matches, in fact.

Other caveman use fire to cook food. Fire scary and hurt. No understand fire. Fire cavemen bad.

Other caveman use nuke to wipe out city. Nuke scary and hurt. No understand nuke. Nuke caveman bad.

Other caveman use anthrax in subway station. Anthrax scary and hurt…

Is AI closer to fire or closer to nukes and engineered viruses? Has fire ever invented a new weapon system?

By the way: we have shitloads of regulations and safety systems around fire due to, you guessed it, the amount of harm it can do by accident.

Re: Language models can explain neurons in language models

#136

Post by bottlepalm »

vhold wrote: post #59
srajabi wrote: post #3

"This work is part of the third pillar of our approach to alignment research: we want to automate the alignment research work itself. A promising aspect of this approach is that it scales with the pace of AI development. As future models become increasingly intelligent and helpful as assistants, we will find better explanations." On first look this is genius but it seems pretty tautological in a way. How do we know i…

It produces examples that can be evaluated. https://openaipublic.blob.core.windows.net/neuron-explainer/...

Using 'im feeling lucky' from the neuron viewer is a really cool way to explore different neurons. And then being able to navigate up and down through the net to related neurons.

Re: Language models can explain neurons in language models

#137

Post by chrisco255 »

PeterisP wrote: post #106
chrisco255 wrote: post #30

Earlier quoted context omitted.

Are there any examples of an LLM developing concepts that do not exist or cannot be inferred from its training set?

Tautologically, every concept that anything (LLM, or human, or alien) develops can be inferred from the input data(e.g. training set), because it was.

No, it wasn't, language itself didn't even exist at one point. It wasn't inferred from training data into existence because such examples existed before. Now we have a dictionary of tens of thousands of words, which describe high level ideas, abstractions, and concepts that someone, somewhere along the line had to invent.

And I'm not talking about imitation nor am I interested in semantic games, I'm talking about raw inventiveness. Not a stochastic parrot looping through a large corpus of information and a table of weights on word pairings.

Has AI ever managed to learn something humans didn't already know? It's got all the physics text books in its data set. Can it make novel inferences from that? How about in math?

Re: Language models can explain neurons in language models

#139

Post by int_19h »

jahewson wrote: post #113
rvz wrote: post #102

Earlier quoted context omitted.

> Which is to say, we still have no clue as to what's going on inside GPT-4 or even GPT-3, which I think is the question many want an answer to. Exactly. Especially: > ...the technique is already very computationally intensive, and the focus on individual neurons as a function of input means that they can't "reverse engineer" larger structures composed of multiple neurons nor a neuron that has multiple roles; This pa…

> It is also the reason why they cannot be trusted in the most serious of applications which such decision making requires lots of transparency rather than a model regurgitating nonsense confidently. Like say, in court to detect if someone is lying? Or at an airport to detect drugs?

You don't even have to look that far ahead. Apparently, people are already using ChatGPT to compile custom diet plans for themselves, and they expect it to take into account the information they supply regarding their allergies etc.

But, yes, those are also good examples of what we shouldn't be doing, but are going to do anyway.

Re: Language models can explain neurons in language models

#140

Post by throwuwu »

ShamelessC wrote: post #118
ftxbro wrote: post #45

I wonder will someone please check the neurons associated to the petertodd and other anomalous glitch tokens ( https://www.lesswrong.com/posts/jkY6QdCfAXHJk3kea/the-petert... )? I can see the github and I see that for any given neuron you can see associated tokens but I don't see how to do an inverse search.

Those were discovered by finding strings that OpenAI’s tokenizer didn’t properly split up. Because of this, they are treated as singular tokens, and since these don’t occur frequently in the training data, you get what are effectively random outputs when using them. The author definitely tries to up the mysticism knob to 11 though, and the post itself is so long, you can hardly finish it before seeing this obvious cr…

The ironic thing about lesswrong is that it’s quite the opposite in some fantastically oblivious ways.
Post reply on HN