Sometimes I think that downvoting, including on websites like Hacker News, is a bad idea. It's easy to see the symmetry with upvotes and think that if you have one, you should probably have the other but psychologically speaking it's a very different type of interaction.
In particular, and even on HN, the good old mantra "don't downvote if you disagree, downvote if the comment doesn't contribute to the discussion" is, from personal experience, not really followed by a significant chunk of the userbase. I try not to care about karma but it's always frustrating when you spend a few minutes of your time carefully composing a comment only to see it grayed out because people didn't like it. It's really toxic too because you don't feel welcome to express your point of view, even if it's in a completely respectful manner. When unchecked this tendency ends up with what you see on most of Reddit: dumb, content-free comments that go with the popular opinion raise to the top while insightful but controversial positions are buried under downvotes. Then you end up with echo chambers.
I wonder how different HN wouldn't be if there were no downvotes at all and only the "flag" button to tag non-constructive comments. Basically the way it works for submissions.
Personally I very rarely downvote comments on HN, if I find a comment that I don't find very insightful but is still technically contributing to the discussion I prefer to find a sibling comment that I deem more interesting and upvote that instead. I turn a negative interaction into a positive one. Instead of focusing on pushing people down you help those you find more interesting.