Even game mechanics have a problem.
Say your valued members have a certain mindset, ethos and style. Let's call it YC-meme. Everyone with YC-meme contributes in a manner that exemplifies YC-meme. If someone comes along and doesn't have that meme, their posts tend to not be what the memesters want. So, members use their voting powers to reduce that person's karma. Eventually they either leave, get ignored by all, or become infected with YC-meme.
Now, open up the community to everyone. At first, YC-meme is still dominant. Only 1% of people without it are able to write posts in a style similar enough to gather karma. Let's say they have RD-meme, which has about 80% overlap with YC-meme. Great.
The problem is, soon, there isn't 1% of the community with RD-meme, there's 10%. Who can (and do) re-enforce RD-meme posts, because it's what they want. People can even get lucky and simply contribute 'good' posts, long enough that they get a decent karma score, and so can re-enforce different memes.
What makes it WORSE is that what's acceptible is what's visible, so with more RD-meme posts, RD-meme posts become more likely, as the probability that they get 'punished' decreases.
End result: Community-dilution runaway.
There are a few 'cures' I can think of, but most are fairly prescriptive:
1. A concrete set of rules and behaviors that anyone can reference to moderate from. Gross.
2. A very high bar for moderation, with active participation. Given that the people most likely to have YC-meme are also likely to be busy, this might be doomed by design.
3. "Proof" required for contribution. Great way to kill off your community.
4. Invitations. Either so easy to get as to be meaningless, or so hard as to stagnate the community.
5. Close ranks to just people at/gone from YC. Sadly, that'd mean I was out. May still be the best alternative.
Maybe the community is self-aware enough to start consciously moderating better. Perhaps a post like this every so often might serve as a gentle reminder. That'd be the best outcome, but needs to be chosen by a majority of the user base. Do YOU want a stricter HN?