Fun fact: I find downvoting to be a completely worthless feature of this site for all of the reasons you've listed and more!
I find it useless to be in possession of the capacity to downvote, as well as the recipient of downvotes from others. This has provoked me to adopt enhanced counter strategies with the full awareness that my behavior is both unorthodox, and contrarian to the culture and preferred/espoused norms/mores of this site as a whole.
In particular, I've noticed that one contrarian comment, which directly contradicts a parent, and indeed violates the typical cultural views of the site as a whole, while remaining on-topic, maintaining valid points, and supporting evidence, receives way more attentive mileage, than any downvote. This is usually because those with vested interests in expressing a typical norm struggle deeply to contradict sound evidence and factual observations.
So, I've stopped maintaining normal accounts, stopped using accounts that have the ability to downvote, stopped downvoting, and instead downvote with words that actually explain my motivations for what would be a downvote, if I were the sort of person to actually resort to petty downvoting.
Instead of pining for the "privilege" of downvoting my peers, I simply unspool evermore single-use accounts and tell people exactly why I disagree with them, point-blank, and never look back, or care about the superficial shade of text coloring my comment, nor the meaningless negative number in a point system on an account that will never be used again.
Downvoters are really only silencing themselves, by refusing to go out on a limb and risk downvotes themselves, rather than express disagreement.