You're thinking good thoughts as far as being aware of the various effects that software design will have on community. Hacker News, for example, restricts certain actions (like downvoting) until you've crossed a certain participation threshold. Other sites disallow downvoting entirely. Some sites don't allow images or links. Others do. Some require a few bucks payment to join. There are all sorts of technical tricks to help tune a community.
But there's also just non-technical stuff. Which I would argue is actually more important, since any piece of social software can be perverted if a certain subset of users is determined. A non-technical example: setting community norms. Hacker News also does this reasonably well -- the community doesn't respond well to goofy memes, for example. This isn't something you read in the rule book, it's something you experience by reading comments and feeling out the tone of the conversation. And if you do post something that doesn't fit the community, you'll pick up a few downvotes or some-such. Most users understand the nature of the community so it self-polices relatively well (with a little help from moderators). I think large open communities (Twitter, Reddit) that need to maintain a ton of traffic to make money have trouble with this: It's tough to want to service the community needs of the entire planet and tell people "Behavior X isn't something that's appropriate here."
I'm not an expert and before you tear me a new one, remember that this is a Hacker News comment and not a peer reviewed journal article. I'm just tossing out some thoughts.
But my belief is that designing for communities is actually quite a challenge, and the fact of the matter is most people putting communities up online right now don't really put much thought into it. Which is a shame. It allows a small subset of toxic users to really damage what should be a good thing for the rest of us.
Back in the days of yore -- late 90s through 2006-ish -- community design was actually a pretty hotly debated topic. But, like I said, it really seems like these days old fashioned comment threads (like you see on news articles), Facebook, Reddit, and Twitter have established themselves as the blueprint for community design and people just sort of copy those without thinking about how they do and don't work. And then online communities become toxic and some people assume all online communities are toxic and then some people start thinking all people are toxic and -- hey! Look where we are. (I'm stretching a bit at the end, here, of course. ;-) )