In your opinion what is the purpose for the upvote and downvote? Upvote = "I'd like to see more comments like this". Downvote = "I'd like to see fewer comments like this". I don't think that people having different reasons for wanting more or less of a particular type of comment is a problem.
I think downvoting should not be. It is an unnecessary evil. A "dichotomy". Fighting for Peace using nuclear power. The best way to fight comments/posts that add no value is to Remain Silent. If you hit someone, there are very likely to hit you back, but if you ignore them they are more likely to take their rants and stories to Digg or somewhere else.