Understanding ChatGPT

> Can you give an example a prompt that shows it does not have meta-awareness and meta-reasoning

Previously here: https://news.ycombinator.com/threads?id=usaar333#35275295

Similar problems with this simple prompt:

> Lily puts her keys in an opaque box with a lid on the top and closes it. She leaves. Bob comes back, opens the box, removes the keys, and closes the box, and places the keys on top of the box. Bob leaves.

>Lily returns, wanting her keys. What does she do?

ChatGPT4:

> Lily, expecting her keys to be inside the opaque box, would likely open the box to retrieve them. Upon discovering that the keys are not inside, she may become confused or concerned. However, she would then probably notice the keys placed on top of the box, pick them up, and proceed with her original intention.

GPT4 cannot (without heavy hinting) infer that Lily would have seen the keys before she even opened them! What's amusing is that if you change the prompt to "transparent", it understands she sees them on top of the box immediately and never opens it -- more the actions of a word probability engine than a "reasoning" system.

That is, it can't really "reason" about the world and doesn't have awareness of what it's even writing. It's just an extremely good pattern matcher.

> Can you give an example of a truly novel problem that it solves worse than a child? How old is the child?

See above. 7. All sorts of custom theory of mind problems it fails. Gives a crazy answer to:

> Jane leaves her cat in a box and leaves. Afterwards, Billy moves the cat to the table and leaves. Jane returns and finds her cat in the box. Billy returns. What might Jane say to Billy?

Where it assumes Jane knows Billy moved the cat (which she doesn't).

I also had difficulty with GPT4 getting it to commit to sane answers for mixing different colors of light. It has difficulty on complex ratios in understanding that green + red + blue needs to consistently create a white. i.e. even after a shot of clear explanation, it couldn't generalize that N:M:M of the primary colors must produce a saturated primary color (my kid again could do that after one shot).

> True, but you can let it use output tokens as scratch space and then only look at the final result. That lets it behave as if it has memory.

Yes, but it has difficulties maintaining a consistent thought line. I've found with custom multi-step problems it will start hallucinating.

> To the contrary, the trend of increasingly large transformers seemingly getting qualitatively smarter indicates that maybe the architecture matters less than the scale/training data/cost function.

I think "intelligence" is difficult to define, but there's something to be said how different transformers are from the human mind. They end up with very different strengths and weaknesses.

Great video indeed. It's very illuminating how gpt works.

Have you tried a prompt like "Write a brief fictional story that answers the question: What is the goop in the gorroborin?" I got as useful an answer as I could expect: the goop is unicorn tears.

"“LLMs are good at single functions, but they can’t understand a system.” This is simply a matter of increasing the context limit, is it not? And was there not a leaked OpenAI document showing a future offering of 64k tokens?"

It's a matter of exponentially increasing complexity, and does the model necessary to create more complex systems have training dataset requirements that exceed our current technology level/data availability?

At some point the information-manipulation ends and the real world begins. Testing is required even for the simple functions it produces today, because theoretically the AI only has the same information as is present in publicly available data, which is naturally incomplete and often incorrect. To test/iterate something properly will require experts who understand the generated system intimately with "data" (their expertise) present in quantities too small to be trained on. It won't be enough to just turn the GPT loose and accept whatever it spits out at face value, although I expect many an arrogant, predatory VC-backed startup to try and hurt enough people that man-in-the-loop regulation eventually comes down.

As it stands GPT-whatever is effectively advanced search with language generation. It's turning out to be extremely useful, but it's limited by the sum-total of what's available on the internet in sufficient quantities to train the model. We've basically created a more efficient way to discover what we collectively already know how to do, just like Google back in the day. That's awesome, but it only goes so far. It's similar to how the publicly traded stock market is the best equity pricing tool we have because it combines all the knowledge contained in every buy/sell decision. It's still quite often wrong, on both short and long-term horizons. Otherwise it would only ever go up and to the right.

A lot of the sentiment I'm seeing reminds me of the "soon we'll be living on the moon!" sentiment of the post-Apollo era. Turns out it was a little more complicated than people anticipated.

> I share ability to move around and feel pain with apes and cats.

You share the ability to predict words with LLMs.

Something being able to do [a subset of things another thing can do] does not make them the same thing.