Thoughtful comments that are factually wrong should be downvoted. Seriously. If your code doesn't compile, the compiler doesn't care how much effort it took you to write it...
It seems more correct to downvote those posts or replies that do not follow the vein of intellectual conversation that HN tries to promote, those comments that distract you from the main topic instead of giving you more info to chew on.
Similarly, those that do make you think, whether you agree or not, have contributed to the conversation and should not be downvoted. If you only wanted correct comments, then this wouldn't be a conversation, it would be going over what you already know, saying out loud what you read in a book or just plain brain masturbation.
Let's have a conversation.