Earlier quoted context omitted.
For me the freakish thing was having GPT-4 complete the prefix of a script, which was mostly a doc-comment and the inputs, written in my style. The completed body of the script was also in my style, even though I had not given GPT any actual "code" in the prompt. It somehow had learned that people that write the preamble "just so" also write script logic in a certain specific way, and it could emulate that.
One of the bigger difficulties with coming to terms with LLMs is that we are either far simpler and easy to model than we once thought, or LLMs are vastly more complex than we comprehend. I am not sure I like either.
(And escaping binary thinking is something humans need to do a better job with in general.)