Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How does GPT obtain its ability? Tracing emergent abilities of language models

yaofu.notion.site

Post reply on HN
Print view View original

131–140 of 205 posts

Re: How does GPT obtain its ability? Tracing emergent abilities of language models

#131

Post by theonemind »

Digory wrote: post #108

This just confirms I'm not minimally competent in this conversation. Is there a "college freshman" explainer? GPT seems to be doing something incredibly different than prior AI. Is it really a Bayesian "next word" chooser at incredible scale?

I want to ask GPT to explain how GPT works using a simple metaphor and common language without computer science or AI field terms, where simplicity is more important than correctness.

Unfortunately, it would probably make up something extremely plausible sounding and very wrong.

Re: How does GPT obtain its ability? Tracing emergent abilities of language models

#132

Post by kfarr »

mettamage wrote: post #72
skybrian wrote: post #47

Earlier quoted context omitted.

Yes, this certainly helps. I find it ironic that you can get somewhat better results for nonfiction by giving it more clues about what it's supposed to be pretending to be. It's always pretending, though.

There’s an actual psychological effect for this as well [1]. Authors + uni are in the link. I forgot the name of the effect, don’t have time to do proper research. [1] https://www.themarysue.com/lab-coats-increase-attention/#:~:... .

> Authors Hajo Adam and Adam Galinsky wanted to explore that nature of what they call “enclothed cognition,” the effect that the clothes we are wearing have on our psychology

Re: How does GPT obtain its ability? Tracing emergent abilities of language models

#133

Post by synu »

rvbissell wrote: post #99
kazinator wrote: post #36

Earlier quoted context omitted.

Reasoning is that which knows that it lacks some necessary knowledge, whereas knowledge isn't aware that it lacks some necessary reasoning.

This has to be one of the most insightful sentences I've ever read.

Would you mind expanding on it a bit? I do sincerely appreciate its pithiness, but curious to read it explained a bit further.

Re: How does GPT obtain its ability? Tracing emergent abilities of language models

#134

Post by lordnacho »

Gigachad wrote: post #82
PartiallyTyped wrote: post #79

Earlier quoted context omitted.

The required knowledge is around entity recognition, with "Obama" referring to the 44th POTUS, and not somebody else who happens to have the same surname (and there are multiple of them actually, at least 4 given his family).

Fair, although I’d think it acceptable if the response to your prompt was “which one? I found 4”

Isn't it a sort of cultural knowledge to understand what is meant? Ie when we say Obama we mean one specific guy who is very important but if I'm talking about Mrs Watanabe it's a generic Japanese person?

Re: How does GPT obtain its ability? Tracing emergent abilities of language models

#135

Post by elcomet »

PartiallyTyped wrote: post #79
Gigachad wrote: post #73

Earlier quoted context omitted.

Doesn't seem to unreasonable to me. If I asked you "How old is Obama" and you had a data source which had the ages of every person, you don't need to know the answer from memory. Your reasoning tells you that to find out how old someone is you need to check the external resource and what to do with the info once you get it.

The required knowledge is around entity recognition, with "Obama" referring to the 44th POTUS, and not somebody else who happens to have the same surname (and there are multiple of them actually, at least 4 given his family).

This can clearly be guessed from a search as well. Popularity can be well defined, and in the case of Obama, there is clearly one much more popular than the others.

Re: How does GPT obtain its ability? Tracing emergent abilities of language models

#137

Post by dunefox »

lovecg wrote: post #130
kazinator wrote: post #36

Earlier quoted context omitted.

Reasoning is that which knows that it lacks some necessary knowledge, whereas knowledge isn't aware that it lacks some necessary reasoning.

Fascinating - makes me think of the distinction between dreaming and being awake. Only in the latter state one can tell the difference.

Not quite true, see lucid dreaming.

Re: How does GPT obtain its ability? Tracing emergent abilities of language models

#139

Post by anon35 »

synu wrote: post #133
rvbissell wrote: post #99

Earlier quoted context omitted.

This has to be one of the most insightful sentences I've ever read.

Would you mind expanding on it a bit? I do sincerely appreciate its pithiness, but curious to read it explained a bit further.

Think of it as: reasoning=computation, knowledge=data. Data alone doesn’t say it must be computed. But computation, by definition, is attempting to create data (the result) that doesn’t exist. Thus: knowledge isn’t aware it must be reasoned about, but reasoning knows it’s trying to find (deduce, compute) knowledge it lacks.

Re: How does GPT obtain its ability? Tracing emergent abilities of language models

#140

Post by fragmede »

resource0x wrote: post #43

Can anyone explain why the GPT response to the same question 4 days ago is different from today's: [ 4 days ago] > Can I learn to sing like Pavarotti? It is possible to learn to sing like Pavarotti, but it will require a lot of practice, dedication, and training. Pavarotti was a renowned opera singer known for his incredible vocal range, powerful voice, and technical mastery. He had many years of training and experie…

There's naturally going to be some instability of its outputs, but I'd bet OpenAI also isn't sitting still. Behind the scenes I'm sure there's a lot of work going on, so the ChatGPT you interacted with on Monday isn't the same one as the one today.
Post reply on HN