Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Language models can explain neurons in language models

openai.com

Post reply on HN
Print view View original

241–250 of 497 posts

Re: Language models can explain neurons in language models

#241

Post by mr_toad »

jacooper wrote: post #34

For people overwhelmed by all the AI science speak, just spend a few minutes with bing or phind and it will explain everything surprisingly well. Imagine telling someone in the middle of 2020, that in three years a computer will be able to speak, reason and explain everything as if it was a human, absolutely incredible!

In 2020 we were headed for an AI winter, according to all the hot takes:

https://www.bbc.com/news/technology-51064369

https://link.springer.com/article/10.1007/s13347-020-00396-6

https://blog.re-work.co/ai-experts-discuss-the-possibility-o...

Re: Language models can explain neurons in language models

#242

Post by NumberWangMan »

PeterisP wrote: post #122
lubesGordi wrote: post #94

Earlier quoted context omitted.

I heard him on Lex too, and it seemed to be just a given that AI is going to be deceptive and want to kill us all. I don't think there was a single example of how that could be accomplished given. I'm open to hearing thoughts on this, maybe I'm not creative enough to see the 'obvious' ways this could happen.

IMHO the argument isn't that AI is definitely going to be deceptive and want to kill us all, but rather that if you're 90% sure that AI is going to be just fine, that 10% of existential risk is simply not acceptable, so you should assume that this level of certainty isn't enough and you should act as if AI may be deceptive and may kill us all and take very serious preventive measures even if you're quite certain that…

And even with all that, probably it's best to still exercise an abundance of caution, because you might have made a mistake somewhere.

Re: Language models can explain neurons in language models

#243

Post by famouswaffles »

ChatGTP wrote: post #202
tschumacher wrote: post #199

Even if we can explain the function of a single neuron what do we gain? If the goal is to reason about safety of computer vision in automated driving as an example, we would need to understand the system as a whole. The whole point of neural networks is to solve nuanced problems we can't clearly define. The fuzziness of the problems those systems solve is fundamentally at odds with the intent to reason about them.

I have to agree. I often think, “maybe I should use ChatGPT for this” then I realise I have very little way to verify what it tells me and as someone working in engineering, If I don’t understand the black box, I just can’t do it. I’m attracted to open source, because I can look at the code understand it.

Open source doesn't mean you can explain the black box any better. and humans are black boxes that don't understand their mental processes either. We're currently better than LLMs at it i suppose but we're still very poor at it.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ https://pure.uva.nl/ws/files/25987577/Split_Brain.pdf https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4204522

We can't recreate previous mental states, we just do a pretty good job (usually) of rationalizing decisions after the fact.

Re: Language models can explain neurons in language models

#244

Post by akiselev »

p1esk wrote: post #219
techolic wrote: post #215

Earlier quoted context omitted.

> that we're pretty damn close to recreating it Is that evident already or are we fitting the definition of intelligence without being aware?

If you spent any time with GPT-4 it should be evident.

If you spent even more time with GPT-4 it would be evident that it is definitely not. Especially if you try to use it as some kind of autonomous agent.

Re: Language models can explain neurons in language models

#246

Post by didericis »

istjohn wrote: post #203
srcreigh wrote: post #182

Earlier quoted context omitted.

What's the argument that understanding neurons is necessary? Perhaps intelligence is like a black box input to our bodies (call it the "soul", even though this isn't testable and therefore not a hypothesis). The mind therefore wouldn't play any more of a role in intelligence than the eye. And I'm not sure people would say the eye is necessary for understanding intelligence. Now, I'm not really in a position to argue…

Why would you doubt neurons play a roll in intelligence when we've seen so much success in emulating human intelligence with artificial neural networks? It might have been an interesting argument 20 years ago. It's just silly now.

> It might have been an interesting argument 20 years ago. It’s just silly now.

Is it?

These networks are capable of copying something, yes. Do we have a good understanding of what that is?

Not really, no. At least I don’t. I’m sure lots of people have a much better understanding than I do, but I think its hard to know exactly whats going on.

People dismiss the stochastic parrot argument because of how impressive big neural nets are, but it doesn’t really invalidate that argument. Is a very, very, very good parrot that learns from everyone at once doing basically the same as what we do? I’d argue no, at least not fully. It’s absorbed aspects of us extremely well/is a very weird, sophisticated mirror, yes, and is copying something somehow, probably in a way reminiscent of how we copy. Is it basically the same as what we’re doing when we think? Partially? Fully? Not at all?

A typical engineer would say “good enough”. That type of response is valuable in a lot of contexts, but I think the willingness to apply it to these models is pretty reckless, even if it’s impossible to easily “prove” why.

To be clear on the exact statement you made, I think you’re right/it’s pretty clear neurons play some very important role/potentially capture a ton of what we consider intelligence, but I don’t think anyone really knows what exactly is being captured/what amount of thought and experience they’re responsible for.

Re: Language models can explain neurons in language models

#247

Post by wilg »

TeMPOraL wrote: post #224
YawningAngel wrote: post #76

Earlier quoted context omitted.

What if you do? LLMs don't have reflexive output or internal streams of thought, they are simply (complex) processes that produce streams of tokens based on an inputted stream of tokens. They don't have a special response to tokens that indicate higher-level thinking to humans.

LLMs seem to me to be the "internal streams of thought". I.e. it's not LLMs that are missing an internal process that humans have, but rather it's humans that have an entire process of conscious thinking built on top of something akin to LLM.

I agree completely and I think this is where a lot of people get tripped up. There's no reason to think an AGI needs to be an LLM alone, it might just be a key building block.

Re: Language models can explain neurons in language models

#248

Post by wilg »

tschumacher wrote: post #199

Even if we can explain the function of a single neuron what do we gain? If the goal is to reason about safety of computer vision in automated driving as an example, we would need to understand the system as a whole. The whole point of neural networks is to solve nuanced problems we can't clearly define. The fuzziness of the problems those systems solve is fundamentally at odds with the intent to reason about them.

> reason about safety of computer vision in automated driving

An interesting analogue. I think we simply aren't going to reason about the internals of neural networks to analyze safety for driving, we're just going to measure safety empirically. This will make many people very upset but it's the best we can do, and probably good enough.

Re: Language models can explain neurons in language models

#249

Post by p1esk »

akiselev wrote: post #244
p1esk wrote: post #219

Earlier quoted context omitted.

If you spent any time with GPT-4 it should be evident.

If you spent even more time with GPT-4 it would be evident that it is definitely not. Especially if you try to use it as some kind of autonomous agent.

What have you tried to do with it?

Re: Language models can explain neurons in language models

#250

Post by hackernewds »

akiselev wrote: post #244
p1esk wrote: post #219

Earlier quoted context omitted.

If you spent any time with GPT-4 it should be evident.

If you spent even more time with GPT-4 it would be evident that it is definitely not. Especially if you try to use it as some kind of autonomous agent.

If you spent even more time with GPT-4 it would be evident that it definitely is. Especialy if you try to use it as some kind of autonomous agent.

(Notice how baseless comments can sway either way)

Post reply on HN