Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Is ChatGPT Doing and Why Does It Work? (2023)

writings.stephenwolfram.com

Post reply on HN
Print view View original

61–70 of 100 posts

Re: What Is ChatGPT Doing and Why Does It Work? (2023)

#61

Post by lxgr »

namaria wrote: post #40
prasoonds wrote: post #15

Earlier quoted context omitted.

This is an interesting thought exercise! The theory goes - that if the model "understands" this scenario, then internally, it has has created something that models the real world. Another interesting bit of experiment people did when the GPT-4 class models launched were to test out spatial awareness. For eg, you could describe with words a construction made of blocks, spheres and so on and then ask questions about th…

Stochastic parrot is way too simplistic a dismissal. It's hand wavy. But I don't get what's so impressive about the nuanced language of a language model that has been given datacenter amounts of compute and virtually all of written word ever put digitalized. Yeah it's the first actually functioning natural language interface. At what cost though. It's completely out of proportion with the benefits and only bubble lev…

Are you referring to the (one-time) training cost or the cost per inferred token? The latter is pretty acceptable these days, especially with smaller models.

Re: What Is ChatGPT Doing and Why Does It Work? (2023)

#62

Post by zeknife »

tga wrote: post #53
Anon84 wrote: post #49

Earlier quoted context omitted.

Here is GPT-4o with a "Please explain your thinking step by step" `The text describes a sequence of events involving two characters, Joe and Sue. Here’s a step-by-step analysis of the text: First Sentence: "Joe drove Sue to university." Action: Joe is driving. Destination: University. Passenger: Sue. Second Sentence: "Afterwards he drove home again and drank a tea with her in the kitchen." Action: Joe drives home. Ti…

A LLM cannot meaningfully “explain its thinking”. When prompted to do so, it will generate an explanation-looking text — but that has nothing to do with the actual way the original answer was generated. If you ask it to explain how a LLM could possibly catch an inconsistency, then it might give you some good insights (or hallucinate something entertaining).

It can't explain it's thinking after producing the answer, like you say, it'll just generate a post-hoc rationalization. But if you ask it to think step by step before making a conclusion, LLM's will somewhat reliably arrive at reasonable steps and better final answers.

Re: What Is ChatGPT Doing and Why Does It Work? (2023)

#63

Post by [deleted] »

mg wrote: post #43
faizshah wrote: post #22

Earlier quoted context omitted.

This is your human interpretation of what it is doing. Another interpretation could be that it has seen similarly structured sentences in other logic/lateral thinking problems so it raises the likelihood of words that have to do with logical inconsistencies and then out of that vocabulary the highest likelihood words are those that come from the nearby context. I would posit that this interpretation is significantly…

You mean a prompt in the form of "A happened, then B happened, then C happened" activates the word "inconsistency"? We can test that by slightly altering the text so that there is no inconsistency anymore and see if the model still claims there is one. We can do so by adding the word "mother" after the word "her": What do you think about the following text? Joe drove Sue to university. Afterwards he drove home again…

[deleted]

Re: What Is ChatGPT Doing and Why Does It Work? (2023)

#64

Post by williamcotton »

faizshah wrote: post #55
mg wrote: post #43

Earlier quoted context omitted.

You mean a prompt in the form of "A happened, then B happened, then C happened" activates the word "inconsistency"? We can test that by slightly altering the text so that there is no inconsistency anymore and see if the model still claims there is one. We can do so by adding the word "mother" after the word "her": What do you think about the following text? Joe drove Sue to university. Afterwards he drove home again…

If you make it a task prompt instead of a lateral thinking task you get a different “world model” or in my view you are just biasing the output to use a different structure. ``` Can you summarize joe’s schedule for my journal? Joe drove Sue to university. Afterwards he drove home again and drank a tea with her in the kitchen. Sure! Here is a summary of Joe's schedule for your journal: --- \*Joe's Schedule:\* - Drove…

Me: Joe drove Sue to university. Afterwords he drove home again and drank a tea with her in the kitchen.

Explain.

ChatGPT: This statement seems to contain an ambiguity regarding Sue's location, suggesting some confusion about the timeline of events. Here are a few possibilities that could clarify the scenario:

1. *Sue Returns with Joe*: - Joe drove Sue to university. - Sue either decided not to stay at university or completed what she needed to do quickly. - Joe drove Sue back home. - They both drank tea together in the kitchen.

2. *Misstatement or Mistake*: - The statement might contain a mistake, implying that Joe and Sue met at home later but doesn't clearly explain how Sue got back from the university. - It could be intended to mean that Joe drove Sue to university and then drove home alone, and Sue somehow returned home later to join Joe for tea.

3. *Joe Visits University*: - Joe drove Sue to university. - Joe stayed at the university, possibly waiting for Sue to finish. - After she was done, they both drove back home together. - They then drank tea in the kitchen.

Based on the original statement, the first scenario seems most plausible, suggesting Sue quickly returned home with Joe.

Re: What Is ChatGPT Doing and Why Does It Work? (2023)

#65

Post by easygenes »

faizshah wrote: post #55
mg wrote: post #43

Earlier quoted context omitted.

You mean a prompt in the form of "A happened, then B happened, then C happened" activates the word "inconsistency"? We can test that by slightly altering the text so that there is no inconsistency anymore and see if the model still claims there is one. We can do so by adding the word "mother" after the word "her": What do you think about the following text? Joe drove Sue to university. Afterwards he drove home again…

If you make it a task prompt instead of a lateral thinking task you get a different “world model” or in my view you are just biasing the output to use a different structure. ``` Can you summarize joe’s schedule for my journal? Joe drove Sue to university. Afterwards he drove home again and drank a tea with her in the kitchen. Sure! Here is a summary of Joe's schedule for your journal: --- \*Joe's Schedule:\* - Drove…

Generally I think this would be a better test if the phrase were “dropped off” Sue at university. As phrased, it’s plausible to assume that Joe stayed at university with Sue, and drove her back and had tea after.

Though a truly smart model should seek to disambiguate situations like this.

Re: What Is ChatGPT Doing and Why Does It Work? (2023)

#66

Post by williamcotton »

hoosieree wrote: post #30
mg wrote: post #5

The better the models get, the harder it is for me to form a mental model of what goes on inside of them. An example of a prompt for which I don't have a good mental model why it works: What do you think about the following text? Joe drove Sue to university. Afterwards he drove home again and drank a tea with her in the kitchen. Older models behaved similar to Markov chains and completely missed that something is log…

> Surely nothing in the prompt directly triggered the word "inconsistency". The prompt is (implicitly) asking to find inconsistencies ("what do you think about") within some statements of fact. Many variations of "find problems in the text" are part of its training set. Remove the "what do you think about" and the model doesn't find inconsistencies. Or keep it, but make the following text more consistent, and watch i…

https://news.ycombinator.com/item?id=40721115

Re: What Is ChatGPT Doing and Why Does It Work? (2023)

#67

Post by williamcotton »

misnome wrote: post #52
mg wrote: post #5

The better the models get, the harder it is for me to form a mental model of what goes on inside of them. An example of a prompt for which I don't have a good mental model why it works: What do you think about the following text? Joe drove Sue to university. Afterwards he drove home again and drank a tea with her in the kitchen. Older models behaved similar to Markov chains and completely missed that something is log…

Does this work without “Clever Hans” prompting it with the implication that there is something to notice?

Yes.

Re: What Is ChatGPT Doing and Why Does It Work? (2023)

#68

Post by dTal »

amelius wrote: post #19
prasoonds wrote: post #15

Earlier quoted context omitted.

This is an interesting thought exercise! The theory goes - that if the model "understands" this scenario, then internally, it has has created something that models the real world. Another interesting bit of experiment people did when the GPT-4 class models launched were to test out spatial awareness. For eg, you could describe with words a construction made of blocks, spheres and so on and then ask questions about th…

> The theory goes - that if the model "understands" this scenario, then internally, it has has created something that models the real world. But this is merely a definition of what it means to "understand" something. For example just tabulating many input/output combinations would not follow this definition.

From the preface to the 20th anniversary edition of Godel, Escher, Bach:

"Meaning cannot be kept out of formal systems when sufficiently complex isomorphisms arise. Meaning comes in despite one's best efforts to keep symbols meaningless! ...When a system of "meaningless" symbols has patterns in it that accurately track, or mirror, various phenomena in the world, then that tracking or mirroring imbues the symbols with some degree of meaning -- indeed, such tracking or mirroring is no less and no more than what meaning is. Depending on how complex and subtle and reliable the tracking is, different degrees of meaningfulness arise."

In other words, when one can reliably ask a language model a question and get a sensible answer, one is forced to conclude that it does in some sense "understand" what it is saying. This is also I think the essential philosophical thrust of the Turing Test, which is often misunderstood as a mere benchmark.

(I notice a common objection to examples of LLMs clearly demonstrating understanding is "it saw something similar in the training set". That may be true (though unfalsifiable) in any given instance, but the number of permutations of things LLMs correctly respond to far exceed the size of any training set. They are certainly generalizing, and interpreting their inputs on a conceptual level.)

Re: What Is ChatGPT Doing and Why Does It Work? (2023)

#69

Post by BoorishBears »

easygenes wrote: post #65
faizshah wrote: post #55

Earlier quoted context omitted.

If you make it a task prompt instead of a lateral thinking task you get a different “world model” or in my view you are just biasing the output to use a different structure. ``` Can you summarize joe’s schedule for my journal? Joe drove Sue to university. Afterwards he drove home again and drank a tea with her in the kitchen. Sure! Here is a summary of Joe's schedule for your journal: --- \*Joe's Schedule:\* - Drove…

Generally I think this would be a better test if the phrase were “dropped off” Sue at university. As phrased, it’s plausible to assume that Joe stayed at university with Sue, and drove her back and had tea after. Though a truly smart model should seek to disambiguate situations like this.

I actually disagree with assuming an inconsistency when no time frame is specified and the phrase isn't "dropped off" as you suggested.

I tried giving Llama 3 70B a prompt that addresses both issues:

```

Can you summarize joe’s schedule for my journal?

4:00: Joe leaves the house with Sue 4:30: Joe drops Sue off at the university 5:00: Joe drinks tea with Sue in the kitchen

```

Reply:

```

I think there may be a problem here!

According to the schedule, Joe drops Sue off at the university at 4:30, but then he's supposed to drink tea with her in the kitchen at 5:00. That doesn't seem possible, since Sue is no longer with him after 4:30.

If you could clarify or correct the schedule, I'd be happy to help you summarize it for your journal!

```

Even Llama 8B can catch this once told `You think out loud before replying going forward`, which imo should be part of any system prompt for an LLM:

```

Let me think out loud for a moment...

Okay, so Joe's schedule starts at 4:00 PM, and he leaves the house with Sue. That's a good starting point.

Hmmm... then at 4:30 PM, Joe drops Sue off at the university. That means Sue is no longer with Joe at this point.

Wait a minute... but then at 5:00 PM, Joe is drinking tea with Sue in the kitchen. That's a bit confusing, since I thought Sue was already at the university.

Ah, I think I see what's going on here. It looks like there's a mistake in the schedule. Joe can't be drinking tea with Sue in the kitchen at 5:00 PM if he dropped her off at the university at 4:30 PM.

```

-

I even tried generating a synthetic schedule with a full day's worth of items: 70B still caught the contradiction, 8B no longer did

Re: What Is ChatGPT Doing and Why Does It Work? (2023)

#70

Post by mistermann »

zeknife wrote: post #56
mg wrote: post #43

Earlier quoted context omitted.

You mean a prompt in the form of "A happened, then B happened, then C happened" activates the word "inconsistency"? We can test that by slightly altering the text so that there is no inconsistency anymore and see if the model still claims there is one. We can do so by adding the word "mother" after the word "her": What do you think about the following text? Joe drove Sue to university. Afterwards he drove home again…

Talking about sentence structure in the conventional sense may not be meaningful here, since what could be described as reasoning in LLM's happens in a more abstract space. If we're looking to understand why a small change makes a big difference, it's pretty intuitive to consider that the second instance of "her" is modified by "mother" due to attention, and ends up being a wildly different vector. Regardless, it's r…

This seems right to me as well: it abstracts the text to a temporal/ spatial object model, and the "What do you think about the following text?" prompt invokes analysis of that model according to prior training examples (as opposed to someone else above prompting it to describe the schedule, in which case it isn't concerned about the logic of the situation).

I would guess that the human mind does this abstraction behind the scenes invisibly, screwing up our intuition when analyzing how LLM's work. I wonder if using examples that are counterintuitive to human intuition might offer insight, because humans reveal their perceived logical thinking is not actually that (rather, is heuristics) in their post-hoc rationalization of the "logic" they believe their mind executed to produce the answer.

(I don't think I articulated what I'm thinking here very well...or, perhaps I have fallen victim to my very own theory!)

A bit more effort...the text is converted into not only tokens, but also abstract tokens, and it is because of the translation into abstract tokens that it is able to match it to training data (which would also have to be translated into abstract tokens). How it resolves the inconsistency after that translation though is beyond me, but it wouldn't surprise me if it is (in this case) a rather trivial problem to someone with depth in logic or some other related discipline.

Post reply on HN