> This has led me to a theory that humans just can't behave nicely beyond some threshold group size.
I think you're generalizing far too broadly. The problem you're describing is more-or-less exclusively a problem with online, open-membership groups.
Consider: if the groups you describe were in-person groups, these ranters would constantly be getting disengaged/off-put/disgusted reactions from the "silent majority" of the people in the group. And just these reactions — together with a lack of any positive engagement — would, almost always, be enough to make them stop or go somewhere else.
(Or, to put a finer point on that: "annoyed, judgemental silence, and then turning away / back to the person you were talking to" would always put off the vast majority of people, with just a few — people who have trouble understanding non-verbal signals — persisting because they aren't "getting the message." And in an in-person context, these few would still eventually be taken aside and given a talking-to, because if they're butting into other in-person conversations with this behavior, they're being far more disruptive than "random new conversation threads" tend to be felt as. Even though "random new conversation threads" can kill a group just as dead.)
The problem with decorum / respect-for-purpose in unmoderated online open-membership groups seems to mostly stem from the fact that people underestimate the importance of non-verbal signals in moderating/regulating behavior. And so there is a dearth of such signals available in such groups. Our brains didn't evolve to play the game of socializing without these signals, any more than ants evolved to coordinate without pheremones. So many people's brains begin to play the game in degenerate / anti-social ways.
From what I've been able to gather, from personal interactions with many people who admit to being "Internet trolls" at some point in their lives... their behavior was almost never intentional maliciousness/active-disregard-for-others on their part. It's rather an emergent behavior — something they "just ended up doing" — given a lack of (non-verbal-signal-alike) calibrating feedback.
And why is there so little non-verbal-signal-alike communication online?
Well, for one thing, we often aren't even aware we're giving off such signals; and so, if we need to consciously choose to communicate them (as we do in online contexts), then we simply fail to do so, because the majority of these signals never even rise to our conscious attention as something to be communicated.
And even when we do become aware of them, we often don't feel them to be important enough to be "worth" going to the effort of translating into some more conscious/explicit/non-subtextual form of communication.
And then, even when a strong desire to communicate a nonverbal signal does bubble up within us... most online chat/forum systems are horrible at transmitting such signals with any degree of fidelity, when they transmit them at all. Especially the kinds of signals used for intra-group behavior regulation.
Facebook, for example, has reaction emojis on both posts and comments — but no reaction emoji that transmits a sentiment like "I disapprove of you saying this; please stop" (e.g. U+1F611 EXPRESSIONLESS FACE or U+1FAE4 FACE WITH DIAGONAL MOUTH). Rather, the only reaction emoji available are those meant to react sympathetically to the emotive content of the post/comment — e.g. with anger, sadness, etc. (People do try to use the "anger" reaction to express disapproval of posts; but when the content itself is often "ragebait" / meant to evoke anger, the poster won't necessarily understand that these reactions are being directed at them, rather than at their post.)
Further, no chat system or forum I'm aware of has participant-visible signals of "detach rate" — i.e. there's no way for people to know when others are clicking on their posts, reading one line, doing a 180 and running away as fast as they can. (YouTube videos expose this metric to their creators; I think it's actually very helpful for them. It could do with being implemented far more widely.)
(And, to be a conspiracy theorist for a moment: I think, in both cases, this is probably intentional. The explicit purpose of signals that "regulate behavior", after all, is to make people engage less in certain anti-social behaviors. Making available any such tools, will therefore inevitably make any kind of platform-aggregate "engagement metrics" go down! If they were ever temporarily introduced, they'd have been quickly removed again with this justification.)