Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How does GPT obtain its ability? Tracing emergent abilities of language models

yaofu.notion.site

Post reply on HN
Print view View original

31–40 of 205 posts

Re: How does GPT obtain its ability? Tracing emergent abilities of language models

#31

Post by vikp »

srajabi wrote: post #2

Amazing insight, particularly section 6. "- The two important but different abilities of GPT-3.5 are *knowledge* and *reasoning*. Generally, it would be ideal if we could *offload the knowledge part to the outside retrieval system and let the language model only focus on reasoning.* This is because: - The model’s internal knowledge is always cut off at a certain time. The model always needs up-to-date knowledge to an…

It's unclear to me how you could separate knowledge and reasoning:

- Reasoning typically requires base knowledge to work from. A side effect of training reasoning is embedding knowledge into the model parameters.

- Even if you offload the search portion (either through outputting special tokens that are postprocessed, or applying the model in multiple steps with postprocessing), you still need embedded knowledge for the model to decide what to search for, and then to successfully integrate that knowledge (in the multi-step case).

Maybe some kind of post-facto pruning of model weights?

Re: How does GPT obtain its ability? Tracing emergent abilities of language models

#32

Post by roywiggins »

nathias wrote: post #7
adamsmith143 wrote: post #4

Earlier quoted context omitted.

The problem with ChatGPT's "knowledge" is that it isn't trustworthy. It will happily output very confident sounding nonsense, or blatantly incorrect statements. We need a way to verify how accurate it's outputs are

so, much like other knowledge sources?

Most knowledge sources don't make up totally fictional citations to nonexistent sources. Or, if they do, nobody uses them for anything serious. Even Wikipedia citations will get removed if they point to URLs that never existed.

Re: How does GPT obtain its ability? Tracing emergent abilities of language models

#33

Post by burnished »

aeternum wrote: post #27
adamsmith143 wrote: post #4

Earlier quoted context omitted.

The problem with ChatGPT's "knowledge" is that it isn't trustworthy. It will happily output very confident sounding nonsense, or blatantly incorrect statements. We need a way to verify how accurate it's outputs are

Whose knowledge is trustworthy? We've somehow come to associate certain institutions or scientific authorities with truth when that is about the furthest from real science: "Have no respect whatsoever for authority; forget who said it and instead look what he starts with, where he ends up, and ask yourself, Is it reasonable?" -Richard P. Feynman "One of the great commandments of science is, "Mistrust arguments from a…

I think part of the issue is that it’s easier to test the limits of or a humans knowledge, and ironically with your quotes I think you’ve supplied evidence that trust is crucial, in that the truest expression of those quotes would be to just deliver the payload and not attach any sort of authority by association to it.

You can’t trust it’s answers (to be fair that’s the existing status quo), but you also can’t easily test it because it will return reasonable sounding garbage. Conversely you can discover ignorance in most humans pretty quickly by exhausting their ability to respond (or your ability to ask).

Re: How does GPT obtain its ability? Tracing emergent abilities of language models

#34

Post by visarga »

adamsmith143 wrote: post #4
srajabi wrote: post #2

Amazing insight, particularly section 6. "- The two important but different abilities of GPT-3.5 are *knowledge* and *reasoning*. Generally, it would be ideal if we could *offload the knowledge part to the outside retrieval system and let the language model only focus on reasoning.* This is because: - The model’s internal knowledge is always cut off at a certain time. The model always needs up-to-date knowledge to an…

The problem with ChatGPT's "knowledge" is that it isn't trustworthy. It will happily output very confident sounding nonsense, or blatantly incorrect statements. We need a way to verify how accurate it's outputs are

You are right, this is the pain point - trust, verification. I think it will become the next focus of research.

There are many things we could do to solve this problem. One of them is to use an external reference for verification. Another one is to train the model to verify facts by augmenting the input with lies - adversarial training for lie detection. Problem solving can be improved by generating more data with the current version of LM for the next one, if we can verify the outputs to be correct.

Re: How does GPT obtain its ability? Tracing emergent abilities of language models

#35

Post by visarga »

burnished wrote: post #33
aeternum wrote: post #27

Earlier quoted context omitted.

Whose knowledge is trustworthy? We've somehow come to associate certain institutions or scientific authorities with truth when that is about the furthest from real science: "Have no respect whatsoever for authority; forget who said it and instead look what he starts with, where he ends up, and ask yourself, Is it reasonable?" -Richard P. Feynman "One of the great commandments of science is, "Mistrust arguments from a…

I think part of the issue is that it’s easier to test the limits of or a humans knowledge, and ironically with your quotes I think you’ve supplied evidence that trust is crucial, in that the truest expression of those quotes would be to just deliver the payload and not attach any sort of authority by association to it. You can’t trust it’s answers (to be fair that’s the existing status quo), but you also can’t easily…

A generative system, be it a neural network or a human, needs a way to test ideas in order to align with reality. If testing is available, then it is possible to advance the state of the art. Ideas are cheap, results matter.

Re: How does GPT obtain its ability? Tracing emergent abilities of language models

#36

Post by kazinator »

vikp wrote: post #31
srajabi wrote: post #2

Amazing insight, particularly section 6. "- The two important but different abilities of GPT-3.5 are *knowledge* and *reasoning*. Generally, it would be ideal if we could *offload the knowledge part to the outside retrieval system and let the language model only focus on reasoning.* This is because: - The model’s internal knowledge is always cut off at a certain time. The model always needs up-to-date knowledge to an…

It's unclear to me how you could separate knowledge and reasoning: - Reasoning typically requires base knowledge to work from. A side effect of training reasoning is embedding knowledge into the model parameters. - Even if you offload the search portion (either through outputting special tokens that are postprocessed, or applying the model in multiple steps with postprocessing), you still need embedded knowledge for…

Reasoning is that which knows that it lacks some necessary knowledge, whereas knowledge isn't aware that it lacks some necessary reasoning.

Re: How does GPT obtain its ability? Tracing emergent abilities of language models

#37

Post by jncraton »

imiric wrote: post #25
page_severian wrote: post #23

Earlier quoted context omitted.

These errors are common and important to note. Though what I find fascinating is how often I can prompt engineer around these errors by telling GPT to just be smarter and think more carefully. Just tried the following for instance. Q: You are a master logician, who can solve all manner of language problems and have excellent math skills. I have 2 cats and 1 horse. How many legs do they have in total? Show your reason…

That's interesting. It seems it's not great at raw calculation, but if you ask it to explain its reasoning, it derives steps from previous results, and arrives at the correct answer. It's similar to the sibling comment about asking it to tell a story.

Correct. That concept is the chain of thought (CoT) reasoning that the article discusses.

Re: How does GPT obtain its ability? Tracing emergent abilities of language models

#38

Post by somrand0 »

seydor wrote: post #20
jimmaswell wrote: post #3

Imagine Wolfram's system with GPT integrated. I hope they're working on that.

they seem to be very different and not at all compatible approaches.

but they're both computation, why wouldn't they compatibly compute?

(I just wanted to say "compatibly computing")

Re: How does GPT obtain its ability? Tracing emergent abilities of language models

#39

Post by visarga »

jtxt wrote: post #14
adamsmith143 wrote: post #4

Earlier quoted context omitted.

The problem with ChatGPT's "knowledge" is that it isn't trustworthy. It will happily output very confident sounding nonsense, or blatantly incorrect statements. We need a way to verify how accurate it's outputs are

One way I tried to do this is by having it write an answer, and a footnote reference at each fact. [1] then list search terms that be used to verify each claim, then I would respond with the url and quotes from found pages for each one, then have it rewrite the answer based on that information and cite the sources. I think something this direction can be automated. I saw someone do this with math and other tasks, tha…

Yes, it's been done both in papers and in various GPT-3 projects. As long as you can find relevant references the LM will become reliable.

Re: How does GPT obtain its ability? Tracing emergent abilities of language models

#40

Post by mensetmanusman »

Is there anything like ChatGPT happening from China?

They have almost 10x’d AI investment compared to the west, and I would be somewhat surprised if that investment combined with ubiquitous math skills failed to produce.

Post reply on HN