I don't think it's as hard to keep a site from sliding as one might think from the examples of previous sites where things went downhill as they got more popular. Slashdot, Digg, and Reddit were all companies. They wanted to grow. Whereas News.YC is a side project. We don't care about growth. It's much easier to do things to keep up the quality when you're willing to sacrifice growth.
As a relative newcomer here, I sincerely hope lolcats are a bannable offense.
What you also need to have is a probation system where you can't post for anywhere from 4 hours to a month depending on what you do. That acts as a warning. Get put on probation too many times and you're banned.
That's how it works at Something Awful, and they still have pretty good quality after almost 10 years. And they are for-profit.
Also, it might be desirable to extra strict during certain times when people are likely to go berzerk and act really stupid (the SA moderators call this the "banhammer" and make a sticky thread notifying users when it is in effect -- what would normally be a warning becomes a ban).