There's a way simpler answer than this Type I Type II thinking stuff. Most LLMs like GPT are not trained on the level of individual characters – they process input and outputs on the level of subword units that compose multiple characters to support long context windows (i.e. "door" instead of "d", "o", "o", "r"). As a result, they do poorly on character manipulation tasks. You can get some insight here: https://beta…
Hm, could explain why it can't play wordle: > Give me a five letter word, beginning with e, and having third letter be t. Not including I, l, u or h. >> etchy > Does etchy contain h? >> No, etchy does not contain h. > which letters do etchy contain? >> Etchy contains the letters e, t, c, h, and y. > what were the criteria? >> The criteria specified that the word must have five letters, begin with e, and have the thir…
It's interesting that it can actually extract the letters from the word despite the tokenization approach.
I have noticed that it struggles with ordinality and cardinality, which might be part of what's going wrong here.
But the other issue with this type of interaction is that once it makes a mistake (claiming etchy meets the criteria), it has a tendency to double down on it and go crazy inventing the reasoning for why.
That's because it's mistake gets baked into the input window and becomes a very strong influence over the output. In this case, you probably would have gotten completely different answers if you informed it etchy was an incorrect answer before asking why it got the answer.