I only downvote when a comment checks all of the following boxes for me: * It has little to no content relating to the conversation. * It is also unnecessarily provocative. Basically if it looks like a troll, walks like a troll, and acts like a troll it get's the downvote.
I try to avoid both upvoting and downvoting based only on whether I agree or disagree with the comment, though I can’t guarantee that I have stuck to this policy rigorously.