All comment scores do is reveal the inner workings of the site. HN already has an algorithm to turn vote count and time since posting into relevance, that's how we order posts now. Perhaps that could be a bit more fine grained, so that you can know more easily to skip barely upvoted leaf nodes, but all seeing the vote score does is let you replace that algorithm with your own internal one.
You are absolutely right though about source links on Google. They give us context for Google's ordering. While Google has done the work to return the best ordering of results possible, which might be the best results in the average case (though maybe not) site have earned a personal reputation for us. Personally, I know that I don't want to check the expertsexchange.com sites, so I skip over them.
I think that what HN would benefit greatly from is a more codified personal reputation system. Right now, the only way we have that is by learning to recognize the names of HN posters which, for me, has only happened with a precious few of the most prolific and respected. If HN would only give me some indication that I had up (or down) voted this commenter n number of times in the past, that would be far more useful than displaying the current vote count. It would allow me to discover whose voices I like, and who I might ignore.
I recognize that not having this is part of the design of HN. Instead we are encouraged to consider every comment fresh, without any context. It's a meritocracy of comments. I don't have much answer to that, other than to say that I would really like to see people be able to earn a good reputation. It would make the community feel a bit smaller and closer.