It's a genuine problem and has been growing gradually worse for a while. I think the cause is simply growth. When a good community grows, it becomes worse in two ways: (a) more recent arrivals don't have as much of whatever quality distinguished the original members, and (b) the large size of the group makes people behave worse, because there is more anonymity in a larger group. I've spent many hours over the past se…
One possible solution (that you might already be doing) is to make the value of the vote proportional to how long the user has been a member, especially down-votes. That solves the problem of more recent arrivals not having the same qualities as the original members. To be very clear so as to prevent conspiracy theories, this is something that we never did on reddit because we didn't want to put that much power into…
Google is the best example to learn from. Search is a misnomer, Google is about ranking. They've put many PhD-centuries of effort into deciding which of the three million matches to put first (ranking). Choosing the three million matches (search) isn't where Google adds its value.
Lots of lessons have been learned there, I'm sure HN can tap into that pool of knowledge by opening up data to the right people.
HN is an awesome resource. I often read the comments before the articles because I expect them to be more useful. It's completely worthwhile to invest a lot of effort in maintaining that greatness as it scales.