It's not surprising that ChatGPT can't play chess well, since:
1) Other than book openings (which it could easily learn) chess requires calculating alternate lines of play, but GPT is just a multi-layer (transformer decoder block) pass-thru architecture so is fundamentally unable to do this. Each word (chess move) output is just the result of N sequential inference steps, with no looping or recursion.
It might potentially be able to do better if prompted/scripted to select few best lines and recursively explore them (using it's own output to keep track of what it is doing), more like a person or computer chess engine would do. Perhaps it could use chess theory to evaluate board positions and which looked good.
2) Even if ChatGPT had the intelligence/reasoning power to learn how to play chess, it'd still need to learn! You can't learn to play chess just passively by studying chess games. It's also in general not memorizing (vs generalizing over) training material, so it's not even going to memorize those games it has seen. Maybe massive fine-tuning on chess would help a bit, but I doubt very much.
3) ChatGPT has tons of knowledge, but a rather limited amount of reasoning power. It's a bit like having a 7-year old idiot savant memorize a set of encyclopedias and then quizzing them on problems you think it should be able to figure out from that. No doubt future models will have better reasoning capabilities and therefore more potential ability to learn something like chess if it also had the necessary iterative planning capability.