I absolutely agree. Invasions are bad, crossing "community" boundaries is. I think that's also part of the "Internet Etiquette" to be learnt: what happens in one domain of the Internet should stay in that domain of the Internet. You don't bring your mischievous trolling behavior from Discord to Facebook, but you also don't bring your "comfortable-space" standards from a support group forum to a video game forum.
I acknowledge your argument that "diverse worlds" system may not be sustainable over time because it is unstable, particularly to new entrants who do have to learn the local lingo. I personally disagree, and would argue instead that that's something Internet users should learn to deal with. We've long had the solution for this: good moderation. You can have hyper-extensive moderation like r/askhistorians or Facebook support groups, you can have moderate moderation for spam and the like for things like generic hobby groups, and then you can have light moderation for spaces deliberately made for that. From heavy to light moderation, the burden shifts from the moderator to the user. Different Internet spaces should have different arrangements like this. While this solution may not be perfect, I argue that this is what works. I put this in contrast to another prevailing idea that all spaces should be like IRL (because that has maximal accountability and requires no context switching), and I argue that that's what will diminish the diversity and dynamism of the Internet.
I would further add a key ingredient to making this world is the tools provided by the platform to perform this moderation. That's one reason why Twitter, for all the good content it has, is also a mess, because there's no limit of cross-pollination of communities. Contrast this to, say, Reddit, which gives subreddit moderators significant powers to curate and protect their communities.