Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Is ChatGPT Doing and Why Does It Work? (2023)

writings.stephenwolfram.com

Post reply on HN
Print view View original

51–60 of 100 posts

Re: What Is ChatGPT Doing and Why Does It Work? (2023)

#51

Post by Kerbonut »

mg wrote: post #5

The better the models get, the harder it is for me to form a mental model of what goes on inside of them. An example of a prompt for which I don't have a good mental model why it works: What do you think about the following text? Joe drove Sue to university. Afterwards he drove home again and drank a tea with her in the kitchen. Older models behaved similar to Markov chains and completely missed that something is log…

I think it's system prompt dependent. I asked LLaMa-2-70B iQ2s which was set up with a system prompt telling it to be logical and detail oriented.

First try:

    I think there is a logical inconsistency in the text. If Joe drove Sue to university, then she was dropped off at the university, which means she would not be available to drink tea with him at his home afterwards. The sequence of events seems to be out of order.
Second try:

    I notice that there's a small issue with the sequence of events. It seems unlikely that Joe drove home again after dropping Sue off at university, only to then drink tea with her in the kitchen. If Joe drove Sue to university, it implies she got out of the car at the university, not that he drove back home with her.

    A more logical sequence of events would be:
    
    Joe drove Sue to university. Afterwards, he drove home and drank a cup of tea by himself in the kitchen.
    
    Or, if you want to keep the scenario where Joe drinks tea with Sue:
    
    Joe drove Sue to university. They went back home together, and then they drank a cup of tea together in the kitchen.

Re: What Is ChatGPT Doing and Why Does It Work? (2023)

#52

Post by misnome »

mg wrote: post #5

The better the models get, the harder it is for me to form a mental model of what goes on inside of them. An example of a prompt for which I don't have a good mental model why it works: What do you think about the following text? Joe drove Sue to university. Afterwards he drove home again and drank a tea with her in the kitchen. Older models behaved similar to Markov chains and completely missed that something is log…

Does this work without “Clever Hans” prompting it with the implication that there is something to notice?

Re: What Is ChatGPT Doing and Why Does It Work? (2023)

#53

Post by tga »

Anon84 wrote: post #49
mg wrote: post #5

The better the models get, the harder it is for me to form a mental model of what goes on inside of them. An example of a prompt for which I don't have a good mental model why it works: What do you think about the following text? Joe drove Sue to university. Afterwards he drove home again and drank a tea with her in the kitchen. Older models behaved similar to Markov chains and completely missed that something is log…

Here is GPT-4o with a "Please explain your thinking step by step" `The text describes a sequence of events involving two characters, Joe and Sue. Here’s a step-by-step analysis of the text: First Sentence: "Joe drove Sue to university." Action: Joe is driving. Destination: University. Passenger: Sue. Second Sentence: "Afterwards he drove home again and drank a tea with her in the kitchen." Action: Joe drives home. Ti…

A LLM cannot meaningfully “explain its thinking”. When prompted to do so, it will generate an explanation-looking text — but that has nothing to do with the actual way the original answer was generated.

If you ask it to explain how a LLM could possibly catch an inconsistency, then it might give you some good insights (or hallucinate something entertaining).

Re: What Is ChatGPT Doing and Why Does It Work? (2023)

#54

Post by accrual »

mg wrote: post #5

The better the models get, the harder it is for me to form a mental model of what goes on inside of them. An example of a prompt for which I don't have a good mental model why it works: What do you think about the following text? Joe drove Sue to university. Afterwards he drove home again and drank a tea with her in the kitchen. Older models behaved similar to Markov chains and completely missed that something is log…

Just to add another sample from another LLM, here is the response from Llama-3-Instruct-8B. I had to specifically ask it identify any possible inconsistency, otherwise it just thought it was a lovely short story.

    > The inconsistency is that Joe drives Sue to university, which suggests it's morning or early afternoon. But then he drives "home again", implying that he was already at his own home before taking Sue to university. This seems unlikely and creates a paradox! What do you think is going on here?

Re: What Is ChatGPT Doing and Why Does It Work? (2023)

#55

Post by faizshah »

mg wrote: post #43
faizshah wrote: post #22

Earlier quoted context omitted.

This is your human interpretation of what it is doing. Another interpretation could be that it has seen similarly structured sentences in other logic/lateral thinking problems so it raises the likelihood of words that have to do with logical inconsistencies and then out of that vocabulary the highest likelihood words are those that come from the nearby context. I would posit that this interpretation is significantly…

You mean a prompt in the form of "A happened, then B happened, then C happened" activates the word "inconsistency"? We can test that by slightly altering the text so that there is no inconsistency anymore and see if the model still claims there is one. We can do so by adding the word "mother" after the word "her": What do you think about the following text? Joe drove Sue to university. Afterwards he drove home again…

If you make it a task prompt instead of a lateral thinking task you get a different “world model” or in my view you are just biasing the output to use a different structure.

```

Can you summarize joe’s schedule for my journal?

  Joe drove Sue to university. Afterwards he drove home again
    and drank a tea with her in the kitchen.

 Sure! Here is a summary of Joe's schedule for your journal:

 ---

 \*Joe's Schedule:\*

 - Drove Sue to university
 - Returned home
 - Drank tea with Sue in the kitchen

 ——-
```

Re: What Is ChatGPT Doing and Why Does It Work? (2023)

#56

Post by zeknife »

mg wrote: post #43
faizshah wrote: post #22

Earlier quoted context omitted.

This is your human interpretation of what it is doing. Another interpretation could be that it has seen similarly structured sentences in other logic/lateral thinking problems so it raises the likelihood of words that have to do with logical inconsistencies and then out of that vocabulary the highest likelihood words are those that come from the nearby context. I would posit that this interpretation is significantly…

You mean a prompt in the form of "A happened, then B happened, then C happened" activates the word "inconsistency"? We can test that by slightly altering the text so that there is no inconsistency anymore and see if the model still claims there is one. We can do so by adding the word "mother" after the word "her": What do you think about the following text? Joe drove Sue to university. Afterwards he drove home again…

Talking about sentence structure in the conventional sense may not be meaningful here, since what could be described as reasoning in LLM's happens in a more abstract space. If we're looking to understand why a small change makes a big difference, it's pretty intuitive to consider that the second instance of "her" is modified by "mother" due to attention, and ends up being a wildly different vector.

Regardless, it's reasonable to assume that certain aspects of the prompt or input structure will prime the model to be more scrutinizing. I'd be surprised to see it point out a logical inconsistency like this if it was just part of a broader context and it wasn't asked "what it thinks" or to "be logical"

Re: What Is ChatGPT Doing and Why Does It Work? (2023)

#57

Post by [deleted] »

BiteCode_dev wrote: post #35
TheGlav wrote: post #34

Earlier quoted context omitted.

I asked the same prompt, but have a memory asking for detailed answers. It gave me: The text presents a logical inconsistency. Here's a breakdown of the issues: Sequence of Events: Joe drives Sue to the university and then returns home. This part is clear. Inconsistency: The text states that after Joe drove home, he drank tea with Sue in the kitchen. However, Sue should be at the university if Joe drove her there, no…

Just yesterday people downvoted me on HN for suggesting ChatGPT had the best ability to deduct things, because, according to down voters, it didn't have any. But the "dumb parrot" narrative that was quite fitting on GPT2, and maybe still useful for understanding the foundation of the tech in GPT3, is now just a demonstration of ignorance. We don't know what they are really doing. Nobody working outside of openai know…

[deleted]

Re: What Is ChatGPT Doing and Why Does It Work? (2023)

#58

Post by [deleted] »

namaria wrote: post #39
mg wrote: post #5

The better the models get, the harder it is for me to form a mental model of what goes on inside of them. An example of a prompt for which I don't have a good mental model why it works: What do you think about the following text? Joe drove Sue to university. Afterwards he drove home again and drank a tea with her in the kitchen. Older models behaved similar to Markov chains and completely missed that something is log…

Occam's Razor says that between the model having somehow developed a persistent internal world model representation and can now reason about it - or is it just big enough in terms of training data, knobs and memory allocation that it can capture more and more language corner cases... Well likely the latter is true.

[deleted]

Re: What Is ChatGPT Doing and Why Does It Work? (2023)

#59

Post by [deleted] »

mg wrote: post #5

The better the models get, the harder it is for me to form a mental model of what goes on inside of them. An example of a prompt for which I don't have a good mental model why it works: What do you think about the following text? Joe drove Sue to university. Afterwards he drove home again and drank a tea with her in the kitchen. Older models behaved similar to Markov chains and completely missed that something is log…

[deleted]

Re: What Is ChatGPT Doing and Why Does It Work? (2023)

#60

Post by danielmarkbruce »

namaria wrote: post #42
danielmarkbruce wrote: post #38

Earlier quoted context omitted.

Stop thinking about words. Think about concepts. As soon as you turn words/tokens into high dimensional embeddings and start playing around with them, they stop being words.

Not when the only way to get anything out of it is using these matrices to line up word tokens. Or play with jpeg noise.

Unclear what you are saying here
Post reply on HN