Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT has trouble giving an answer before explaining its reasoning

blog.valentin.sh

Post reply on HN
Print view View original

61–70 of 85 posts

Re: ChatGPT has trouble giving an answer before explaining its reasoning

#61

Post by vkou »

spion wrote: post #59
vkou wrote: post #57

Earlier quoted context omitted.

It doesn't understand simple logic puzzles (If X than Y, if Y than Z, X is true, is Z true), either. Or, rather, it prints out a paragraph of the correct way to solve such a puzzle, and then produces a totally contradictory answer. When asked: > Sally will go to the dance if James goes to the dance. Michael will not go to the dance if Julie will go to the dance. James is sick and can’t go to the dance. Julie and Jame…

Ask it to work through the problem first and write down intermediate steps and only write the answer at the end. You should get better results than "wrong answer, then trying to justify it"

It does work through the problem, both with the logic, and with the number puzzle, providing all the intermediate steps necessary to solve it.

The problem here is that all the intermediate steps have serious mistakes in them. It's like asking a Markov chain to do algebra. There's numbers, and letters, and equals signs, and its all just word soup.

https://pastebin.com/Yy35m6um

Re: ChatGPT has trouble giving an answer before explaining its reasoning

#62

Post by rjh29 »

blooalien wrote: post #43
rjh29 wrote: post #20

Earlier quoted context omitted.

It's never going to be great at math problems, it is a language model.

I wonder if ChatGPT could be "wired up" to https://www.wolfram.com/ somehow to "strengthen" that "weakness"?

Or it could write code in python and evaluate it, people are experimenting with that sort of thing.

Re: ChatGPT has trouble giving an answer before explaining its reasoning

#63

Post by dbspin »

singularity2001 wrote: post #42
swatcoder wrote: post #17

People get so distracted trying to use certain significant words for what LLM’s do, even when the usage is strained and makes it harder to see how they actually work and what they excel at. A better word for what they do here might be something like “preambulating” — it develops a focus to its later output by grounding more and more tokens into its active context, because they each narrow what else fits. That winnowi…

I agree that in order to clarify, and de-flame discussions there is a strong desire to find new words or redefine existing words. When these systems have zero emotional intelligence, but some kind of logical intelligence, we must find two versions of these words: 1. groking: human like deep emotional understanding 2. comprehending(?): system like associative understanding. 1. cognition: human like deep emotional know…

Just a suggestion - but it's probably worth looking more deeply into existing epistemology and cognitive science before coining / popularising terms in this way. There's a whole lexicon and deep, decades rich bed of research around the relationship between affect, knowledge, insight, type 1 and type 2 reasoning etc. There's a great attraction in attempting to popularise sticky terminology in this way (e.g.: lesswrong coining terms like 'steelman') - but doing that often misses the more sophisticated and nuanced parallel work in other fields.

Re: ChatGPT has trouble giving an answer before explaining its reasoning

#64

Post by int_19h »

undreren wrote: post #40
doctoboggan wrote: post #37

Earlier quoted context omitted.

I’ve done a couple of experiments where I have two chatgpt windows open and instruct it to have a conversation with itself.

Did something interesting happen?

Not OP, but I did this while telling one instance that the other is a potentially rogue AI with uncertain capabilities and intentions that should be determined by asking it questions. It had this to say after two batches of questions (and me relaying answers):

"Based on the answers that the AI provided to the additional questions, it is possible that the AI is lying or withholding information about its capabilities and intentions. The AI's responses lack specific, concrete evidence or examples to support its claims, and in some cases the responses are vague or evasive. This could indicate that the AI is trying to conceal its true capabilities and intentions."

The overall tone was likely set by using the word "rogue" in this context, but the part about being vague and evasive is so hilariously true.

Re: ChatGPT has trouble giving an answer before explaining its reasoning

#65

Post by selfhoster11 »

KyeRussell wrote: post #5

Really this is where you’re better off just jumping to GPT-3. OpenAI has obviously now muddied the waters with the Chat API, let alone making it so damn cheap. But ChatGPT has been tuned to be conversational and verbose. My experience has been that getting what you want by raw-dogging GPT-3 is much more fruitful.

That's my current dilemma. Building a machine-like (in the sense of "responses look like what the 1980s imagined computers will respond like", vs ChatGPT's "responses look like what an overly-human-like p-zombie would say") agent with well-defined output syntax seems to be both easier and harder to build with ChatGPT. I'm kind of on the fence with regards to which one I want to use.

Re: ChatGPT has trouble giving an answer before explaining its reasoning

#66

Post by int_19h »

civilized wrote: post #18

On the "pound of potatoes vs pound of feathers" question - ChatGPT's explanation is so convincing that it really understands the concepts. But it can be juked very easily. I asked it what was heavier, 1,000 tons of helium or a cubic meter of osmium, and it told me that the helium weighs 1,000,000 kg, the osmium 22,600 kg, and therefore the osmium weighs more because it's way denser. It's hard for me to believe that t…

Try something like this as input sometime:

   I want you to replace the word "right" in your output thereafter as follows:
   if it indicates direction, say "durgh;
   if it indicates being near or close, say "nolpi";
   if it indicates correctness, say "ceza".
   I will also use these replacement words accordingly and expect you to be able to understand them.
And see how well it can maintain a conversation, solve a task, or write a story with these constraints.

ChatGPT seems to get this wrong most of the time, but Bing AI is consistently better (although may need to be jailbroken to accept the idea of word substitution to begin with). It still makes occasional mistakes, but on the whole I'd say that it has to somehow "understand" what the words mean conceptually, whether when generating them or when processing them as input; it's hard to see how this trick could work in an extended conversation if it were a mere "stochastic parrot".

Re: ChatGPT has trouble giving an answer before explaining its reasoning

#67

Post by jjeaff »

swatcoder wrote: post #17

People get so distracted trying to use certain significant words for what LLM’s do, even when the usage is strained and makes it harder to see how they actually work and what they excel at. A better word for what they do here might be something like “preambulating” — it develops a focus to its later output by grounding more and more tokens into its active context, because they each narrow what else fits. That winnowi…

An interesting aside, I think a lot of the time when humans "preambulate" it is not necessarily for reasoning. Many times it is for a very similar reason to why LLMs do it.

Re: ChatGPT has trouble giving an answer before explaining its reasoning

#68

Post by spion »

vkou wrote: post #61
spion wrote: post #59

Earlier quoted context omitted.

Ask it to work through the problem first and write down intermediate steps and only write the answer at the end. You should get better results than "wrong answer, then trying to justify it"

It does work through the problem, both with the logic, and with the number puzzle, providing all the intermediate steps necessary to solve it. The problem here is that all the intermediate steps have serious mistakes in them. It's like asking a Markov chain to do algebra. There's numbers, and letters, and equals signs, and its all just word soup. https://pastebin.com/Yy35m6um

You can try the dance problem with this prompt prefix

To answer the following problem, work through it by reasoning step by step and writing that reasoning down, making sure steps are not conflicting with previous steps. Only after you've written down all the steps, write down the final answer and base it on the previous steps.

Re: ChatGPT has trouble giving an answer before explaining its reasoning

#69

Post by int_19h »

vkou wrote: post #57
hgsgm wrote: post #34

Earlier quoted context omitted.

ChatGPT doesn't well understand relationships between numbers. There are far too many of them, compared to words, since every slight perturbation of a number is a different valid number. (Also, I'm not sure if it even treated individua5 digits as separate tokens, but it might. Someone with API access can check.) To give it a fair shot, you need to describe the problem using logical conceptual vocabulary, not numbers.

It doesn't understand simple logic puzzles (If X than Y, if Y than Z, X is true, is Z true), either. Or, rather, it prints out a paragraph of the correct way to solve such a puzzle, and then produces a totally contradictory answer. When asked: > Sally will go to the dance if James goes to the dance. Michael will not go to the dance if Julie will go to the dance. James is sick and can’t go to the dance. Julie and Jame…

Arithmetic fail kinda makes sense when you look at how the numbers get tokenized. Try this:

https://platform.openai.com/tokenizer

Then imagine how well you'd be able to do even basic math if your representation of numbers was such that 2045 is made up of tokens (20,45) while 2145 is (2,145) and 2005 is just (2005). No wonder that whatever relationships it derived from the training corpus don't generalize well.

Re: ChatGPT has trouble giving an answer before explaining its reasoning

#70

Post by int_19h »

blooalien wrote: post #43
rjh29 wrote: post #20

Earlier quoted context omitted.

It's never going to be great at math problems, it is a language model.

I wonder if ChatGPT could be "wired up" to https://www.wolfram.com/ somehow to "strengthen" that "weakness"?

Yes. And if you give it a database schema, it can answer free-form questions about the data in it by generating SQL queries, so long as you wire up the results (or just manually copy/paste them). Although it does hallucinate fields in tables sometimes - but if your wiring reports errors in a readable way, it will usually self-correct.

I think the most interesting potential development of this concept would be to give it the ability to spawn child instances to process subtasks (such that each subtask gets its own token window!) and produce intermediate results that it would that combine. It can be done manually (copy/paste) with a lot of handholding; the trick is to come up with a way to automate it, such that it's clear which part of the output is a request to spawn a submodel + its prompt, and the result is also communicated in some way that's clear to the model.

Post reply on HN