That's indeed interesting. By the way, it found another pattern: [d, f, h] => j. Your starting letters were sequentially meaningful too. According to this pattern the word "jazz" would be acceptable.
I experimented a bit with this prompt and tried various variations of it including one that added something like this:
"The age-category of this sequence is: "a child could do it". You are to solve this by pretending to be a young human with average skills and perceptions. You should assume average common sense in your answer."
Outputs are like this:
=====
"Now, let's look at the words. I noticed that the words have something in common with the numbers that come right before them. They are related to the numbers in this way:
'done' - It sounds like "one" when you say it aloud.
'free' - It sounds like "three" when you say it aloud.
'hive' - It sounds like "five" when you say it aloud."
"So, the pattern in the sequence is alternating between even numbers and words that rhyme with the odd numbers. Here's the whole pattern:
0 (even number), 'done' (rhymes with 'one'), 2 (even number), 'free' (rhymes with 'three'), 4 (even number), 'hive' (rhymes with 'five')"
=====
I agree with you that language models have vastly different failure modes than humans. Often extremely weird, but to even get close to their potential you do have to probe a bit deeper than just asking it stuff without context.
It just semi-randomly walks some semantic space and not all of those branches are fruitfully related to our initial problem. It doesn't know at all what we are doing, there is nothing going on "under the hood".
Edit: to be completely clear I'm not arguing it "does know" this answer or something. I don't know if it even "knows", but I do know a thing or two how to get better answers. Current models still are very clearly limited so sooner or later you'll hit it the limit and they start what I call "averaging": you'll get blurry pictures of what you want.