Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT has trouble giving an answer before explaining its reasoning

blog.valentin.sh

Post reply on HN
Print view View original

71–80 of 85 posts

Re: ChatGPT has trouble giving an answer before explaining its reasoning

#71

Post by taink »

hgsgm wrote: post #34
civilized wrote: post #18

On the "pound of potatoes vs pound of feathers" question - ChatGPT's explanation is so convincing that it really understands the concepts. But it can be juked very easily. I asked it what was heavier, 1,000 tons of helium or a cubic meter of osmium, and it told me that the helium weighs 1,000,000 kg, the osmium 22,600 kg, and therefore the osmium weighs more because it's way denser. It's hard for me to believe that t…

ChatGPT doesn't well understand relationships between numbers. There are far too many of them, compared to words, since every slight perturbation of a number is a different valid number. (Also, I'm not sure if it even treated individua5 digits as separate tokens, but it might. Someone with API access can check.) To give it a fair shot, you need to describe the problem using logical conceptual vocabulary, not numbers.

> Also, I'm not sure if it even treated individual digits as separate tokens, but it might. Someone with API access can check.

Anyone can check, they have a tool for that[1]. It's mentioned in their FAQ article[2].

According to their tool, GPT-3 counts the following as one token:

- any combination of or below 3 digits

- 1111, 3333, 6666, 9999 (it tends to group other digits in groups of 2)

- 66666666 (so 8 sixes -- 5, 6 or 7 won't work)

- 00000000 (anything below 8 zeros counts as one token as well, probably to handle millions and billions)

- 0000000000000000 (16 zeros)

This isn't an exhaustive list, there are probably a lot of other weird edge cases I haven't tried. Its failure to understand basic arithmetic makes much more sense given how inconsistent the tokenizing of digits is done.

[1]: https://platform.openai.com/tokenizer

[2]: https://help.openai.com/en/articles/4936856-what-are-tokens-...

Re: ChatGPT has trouble giving an answer before explaining its reasoning

#72

Post by vkou »

spion wrote: post #68
vkou wrote: post #61

Earlier quoted context omitted.

It does work through the problem, both with the logic, and with the number puzzle, providing all the intermediate steps necessary to solve it. The problem here is that all the intermediate steps have serious mistakes in them. It's like asking a Markov chain to do algebra. There's numbers, and letters, and equals signs, and its all just word soup. https://pastebin.com/Yy35m6um

You can try the dance problem with this prompt prefix To answer the following problem, work through it by reasoning step by step and writing that reasoning down, making sure steps are not conflicting with previous steps. Only after you've written down all the steps, write down the final answer and base it on the previous steps.

I tried the dance problem, and regenerated the response three times.

The first two claimed that Michael will go to the dance, but third one made the correct argument that Michael may or may not go to the dance.

It didn't help it do any better on the number problem. Prepending that paragraph still has it get the first step is wrong, as well as everything that follows it.

> To solve this puzzle, we need to find the values of A, B, and C that satisfy the equation:

> AAA + BBB = AAAC

> Let's start by looking at the rightmost digit, which is C. We know that C must be either 0 or 1 because the sum of two digits cannot be greater than 18 (9 + 9 = 18). Also, C cannot be 0 because that would mean that A and B would be equal, which is not allowed in this puzzle. Therefore, C must be 1.

... And then it keeps going into la-la land.

The final answer it gives is, by the way:

957 + 483 = 1440

Re: ChatGPT has trouble giving an answer before explaining its reasoning

#74

Post by bootsmann »

huevosabio wrote: post #27
jtbayly wrote: post #26

Earlier quoted context omitted.

Is there any reason we cannot let ChatGPT “talk to itself” for a bit before spitting out an answer to us?

Came back to this post after thinking for a while precisely to mention this. Right now, ChatGPT is sort of forced to "think" and talk at the same time, so it's hard for it to "reason" ahead of answering. But, if we allowed him to produce some tokens in silence prior to answering, perhaps it could give even better answers.

> But, if we allowed him to produce some tokens in silence prior to answering

Depending on how the model is implemented this is already the case. Transformers just predict the next token but usually we don't just greedily pick the most likely next token as doing this produces cases where the model just repeats the same sentence or spams tokens it really likes (the enter key). Some more sophisticated techniques, like beam search, produce a different sequences of tokens and try to maximise the score across all tokens in the sequence.

Re: ChatGPT has trouble giving an answer before explaining its reasoning

#75

Post by soheil »

swatcoder wrote: post #17

People get so distracted trying to use certain significant words for what LLM’s do, even when the usage is strained and makes it harder to see how they actually work and what they excel at. A better word for what they do here might be something like “preambulating” — it develops a focus to its later output by grounding more and more tokens into its active context, because they each narrow what else fits. That winnowi…

> But it’s not reasoning

Can you define what reasoning is if not the ability to go from A to B, something that most basic computers can do (e.g., ALU). Granted LLM's do probabilistic reasoning but I don't see that as a seismic shift away from good ol' fashion reasoning which requires much less computation anyway.

Re: ChatGPT has trouble giving an answer before explaining its reasoning

#76

Post by civilized »

int_19h wrote: post #66
civilized wrote: post #18

On the "pound of potatoes vs pound of feathers" question - ChatGPT's explanation is so convincing that it really understands the concepts. But it can be juked very easily. I asked it what was heavier, 1,000 tons of helium or a cubic meter of osmium, and it told me that the helium weighs 1,000,000 kg, the osmium 22,600 kg, and therefore the osmium weighs more because it's way denser. It's hard for me to believe that t…

Try something like this as input sometime: I want you to replace the word "right" in your output thereafter as follows: if it indicates direction, say "durgh; if it indicates being near or close, say "nolpi"; if it indicates correctness, say "ceza". I will also use these replacement words accordingly and expect you to be able to understand them. And see how well it can maintain a conversation, solve a task, or write…

I agree. There are many indicators that it has some sort of deeper understanding of the meaning of language. Even in the conversation I had, for all its flaws, it was able to correctly perceive inconsistencies in its statements based on my prompts and make somewhat coherent attempts to correct them. It's just that the understanding can be so fragile, and its attempts to resolve inconsistencies are superficial, incurious, bullshitty.

"Density matters for weight but not mass" is a perfect example - it's ridiculous, but I can understand how it logically inferred that from its own previous statements. I'd bet plenty of money that it didn't get this crazy idea from its training data.

To be fair, humans have the same sort of issue sometimes. But ChatGPT seems to have more extreme versions of the issue and perseveres confidently with no self-awareness.

Really though, not bad for an autoregressive text model trained on terabytes of internet data.

Re: ChatGPT has trouble giving an answer before explaining its reasoning

#77

Post by leobg »

Darthy wrote: post #28
jtbayly wrote: post #26

Earlier quoted context omitted.

Is there any reason we cannot let ChatGPT “talk to itself” for a bit before spitting out an answer to us?

Here's a thought: If you let ChatGPT be idle, thinking to itself, dreaming and planning, this might actually cross the boundary towards sentience - of what we would call somebody who is alive. So there might be some safety and moral concerns.

What an interesting thought:

What makes us intelligent beyond machines is our time spent silently and introspectively thinking and dreaming in the absence of outside prompts?

If this is true, then we are certainly getting closer to an AI that surpasses us. Because while the AI might start to introspect, we humans gradually do it less and less, given that we are surrounding ourselves with more and more external prompts (information, overload, notifications, TikTok, HN…).

Re: ChatGPT has trouble giving an answer before explaining its reasoning

#78

Post by Llamamoe »

visarga wrote: post #55
civilized wrote: post #18

On the "pound of potatoes vs pound of feathers" question - ChatGPT's explanation is so convincing that it really understands the concepts. But it can be juked very easily. I asked it what was heavier, 1,000 tons of helium or a cubic meter of osmium, and it told me that the helium weighs 1,000,000 kg, the osmium 22,600 kg, and therefore the osmium weighs more because it's way denser. It's hard for me to believe that t…

Works if you ask it to generate a Python program to solve the task. > A cubic meter of osmium is heavier than 1000 tons of helium. > Here's a Python program to print the response: mass_of_helium = 1000 * 1000 # in kilograms density_of_osmium = 22590 # in kilograms per cubic meter volume_of_osmium = 1 # in cubic meter mass_of_osmium = density_of_osmium * volume_of_osmium # in kilograms if mass_of_osmium > mass_of_heli…

So basically ChatGPT has zero executive function and needs a human to plan HOW it uses its "cognitive" capacity in order to get anything right.

Re: ChatGPT has trouble giving an answer before explaining its reasoning

#79

Post by mmusson »

jtbayly wrote: post #26
swatcoder wrote: post #17

People get so distracted trying to use certain significant words for what LLM’s do, even when the usage is strained and makes it harder to see how they actually work and what they excel at. A better word for what they do here might be something like “preambulating” — it develops a focus to its later output by grounding more and more tokens into its active context, because they each narrow what else fits. That winnowi…

Is there any reason we cannot let ChatGPT “talk to itself” for a bit before spitting out an answer to us?

For certain types of prompts like basic arithmetic that would give ChatGPT trouble, it actually does do a good job coming up with useful questions when prompted. For example, if you ask it the answer to 2 * diameter of the Moon, and then give it a prompt such as what information it would need to answer, it will do a good job breaking down the parts of the problem. So there’s no reason that it couldn’t take some prompt like that and turn around and generate queries to gather those facts in order to create the final answer. In this scenario is really chatting with search engines and things like Wolfram Alpha.

Re: ChatGPT has trouble giving an answer before explaining its reasoning

#80

Post by archon1410 »

Llamamoe wrote: post #78
visarga wrote: post #55

Earlier quoted context omitted.

Works if you ask it to generate a Python program to solve the task. > A cubic meter of osmium is heavier than 1000 tons of helium. > Here's a Python program to print the response: mass_of_helium = 1000 * 1000 # in kilograms density_of_osmium = 22590 # in kilograms per cubic meter volume_of_osmium = 1 # in cubic meter mass_of_osmium = density_of_osmium * volume_of_osmium # in kilograms if mass_of_osmium > mass_of_heli…

So basically ChatGPT has zero executive function and needs a human to plan HOW it uses its "cognitive" capacity in order to get anything right.

Executive function, and correctly understanding how to use your cognitive capacity to its full potential, is not completely innate to any person; people need some schooling to learn it, it is something taught. I'm not sure what this has to do with LLMs, but I wonder how many people could multiply even two random two-digit numbers in their head if they weren't taught how to multiply on paper using very simple algorithms.
Post reply on HN