There’s also the Goodhart-flavored line of argument here: at the moment, part of the sales pitch is that these internal conversation channels provide a meaningful view into staff’s reactions, dynamics, and states of mind.
Fire the first few folks for their AI-detected “toxicity,” and one imagines the wiser of their “toxic” friends will find other avenues for their bad behavior. Use sentiment analysis to sound out and reward the true believers in the corporate party line, and suddenly everyone will be Slacking effusive praise for the policies of the dear leaders.
The usefulness of this tech would seem to depend a lot on how the place is managed. The obvious worry is that the powers that be will smoke out dissenters, reward toadyism, and punish harmless individual differences. But I can also imagine a constructive management environment, where the “mothership” worries that their feedback from the line staff is getting distorted through layers of management sucking-up and bureaucracy; and imagining this type of analysis as some kind of independent finger on the pulse of the workforce. That seems like it might be useful for making genuinely good decisions.
Knowing a lot of true stuff and using it to rule out bad guesses we made based on false impressions, that doesn’t seem so bad to me.
Using what you know about an underling to punish them, police them, dehumanize them into nothing more than a pile of scores—that’s neither a new strategy nor one that leads to success.
History seems to show that, while that kind of excess may enforce ideological consistency in the near term, it’s intrinsically costly to the health of the [corporation|country|institution]. Over-condition on these signals, like any others, and you just produce a culture of performative compliance and doublespeak-you make the signal useless. And that leaves you in the same anemic information environment that you started in, the sort that tends to conceals problems until they become overwhelming.