Of course, I dislike being downvoted. If I think it was because I was misunderstood or misinterpreted, I will post in response to the downvote. Clarification sometimes gets the original post upvoted, as well as the clarification. If it's a case of "Yeah, I was trying to avoid getting into that because I just knew it would go over badly and now I've been downvoted, confirming what I thought" or a case of me just not saying something well because I was having a bad day, I generally leave it alone. No point in pouring gasoline on the fire, so to speak (which is generally what's going to happen if I am already sticking my feet in my mouth because I don't feel well and now I am all defensive and trying to "clarify"....downward spiral ensues).
On the one hand, I often do wish people would converse with me and tell me what they "heard" that they think it's a bad comment. In many cases, it turns out be a misunderstanding. The vast majority of meaning in face-to-face communication comes from facial expression, body language, voice tone and context. The actual words are a very small part of the meaning conveyed. But on the 'Net, the words are about all we have. If two people come from very different backgrounds, misunderstandings are practically guaranteed.
On the other hand, an Iranian friend once told me that there is a saying in Iran: "Silence is the only good answer to stupid people." I would modify that slightly to "Silence is the only good answer to something stupid." We all do stupid stuff at times. It isn't proof we are generally stupid. It's just part of the human condition. An anonymous downvote, though it may be upsetting to the receiver, is probably a lot less destructive to the community than being able to put a name to the criticism and having to try to express the criticism in words. Running battles between two members who just don't ever agree on anything can be very destructive to a forum. I think the capacity to downvote helps avoid such outcomes.
I am still getting a feel for how HN works. But so far I am finding that the system here seems to do a better job of minimizing some of the negative social crapola you find everywhere. Conflict of interest will never go away. Interpersonal friction between some members is inevitable. Spammers and all kinds of other undesirable stuff is also a fact of life. There has to be a means to address that. Warm-fuzzy 'let's only upvote people' solutions won't work. It has no teeth.