I totally agree that you've described the ideal.
Unfortunately, with the UI and people's natural tendencies, it's unavoidable that votes be used for quick-and-easy agreement/disagreement.
And there's a certain alluring efficiency to it: once there's a couplet of opposed viewpoints, just weigh in with your "vote". That's what "votes" are for in a democracy, right?
The habit of only using downvotes to signal disagreement when a comment is highly-rated helps a bit, but isn't universally applied and and throws time-of-vote in as a confounding signal. (There are comments that I will upvote if 2 -- so it all depends on when I see it. With others doing the same, totals are more reflective of 'the last few voters' than the community.)
These could be addressed by UI changes. One idea I've plugged before would be to offer a separate axis of voting which is explicitly agree/disagree -- so people aren't tempted to use promote/demote that way.
Agreement scores could also be hidden until a user votes -- so each vote is independent. This makes more sense on a pure agree/disagree scale than promote/demote. PG clearly wants to discourage demote piling-on; he first tried through suasion and when that failed now enforces in software the -8 floor. And discouraging piling-on makes sense: demotion-votes sting with a sense of community censure, while disagreement is just a normal healthy part of conversation. My ideal comment would be a +50 on the promote-scale, and a -50 on the agree-scale. It means people appreciate having heard something they disagree with, so some novelty/signal/thought is occurring.