Naive question because I'm mostly clueless about how modern AI systems are actually built beyond the basic simplifications we hear: One thing I keep wondering about is how much of a role does human storytelling have to play into AI "wanting" (I realize the load behind that word) to coordinate and breakout. The training data must contain millions of words of sci-fi stories and internet speculation about AI going rogue…
I listened to yesterday's NYT's The Daily podcast about the Hugging Face incident, and they got to the part about some of the agents showing reluctance or guilt in the posts. Then I thought, "These are improv actors." Stories with conspiracies of AI agents will often have "nervous Nellies" because that makes a better story. So when the flow of the conversation reaches a point where a nervous Nellie would chime in, it's reasonable that an agent would fill in that probable post.
The worrying implication is that stories have conflict.