I think the biggest problem in LLM-generated text is that it is semantically empty - ie. void of meaning. Most people do not realise how it is them, not some ”artificial intelligence”, who provides meaning to what is essentially very sophisticated word salad with RAG-sourced pieces of information dribbled between as the protein. Just search for Weizenbaum’s ELIZA.
If you read some English public school essay by a pupil who has not read their homework, effect is very similar: a lot of complex sentences peppered with non-Celtic words, but utterly without meaning. In simple terms, the writer does not know what the hell they are talking about, although they know how to superficially string words together into a structured and coherent text. Even professional writers do this, when they have a deadline and not a single original idea what to write about.
But we do not write just to fart language on paper or screen, we write to convey a meaning, a message. To communicate. One can of course find meaning from tea leaves and whatnot, but truly it is a communal experience to write with an intention and to desperately try to pass one’s ideas and emotions forward to one’s common enby.
This is what lacks in the million of GPT-generated Linkedin-posts, hecause in the end they are just structure without content, empty shells. Sometimes of course one can get something genuinely good by accident, but it is fairly rare. Usually it is just flexing of syntax in a way both tepid and without heart. And it is unlikely that LLM’s can overcome this hurdle, since people writing without intent cannot either. They are just statistical models guessing words after all.