I get the desire for answers to be evaluated on their own merits, but here is an argument for the system staying the same as it currently is:
1. If you're asking for help on a topic, then--by definition-- you are in an information asymmetric situation. You don't know what the right answer is. There's a decent chance that you, as the asker of the question, are not in a position to evaluate the answer on its merits.
2. Therefore you have to choose the answer based on other factors. On SO you can do something like see if the proposed answer makes sense to you, and check if the answer produces the results you want. Those are pretty okay metrics. But, obviously, they aren't perfect. Whether it makes sense to you doesn't fully track the truth of the situation, and whether it works or not doesn't address whether it's a good idea for other reasons, like being grossly inefficient, or deprecated, or unmaintainable, or whatever. So there's a decent chance that you can't evaluate the answer on its own merits, AND that sensible heuristics will fail you because you don't know what you don't know about the full context of the question, and might choose a wrong answer as a result.
3. Therefore, an additional useful signal you can track is the reputation and apparent expertise of the person who is answering. Of course, exactly as you say, it's not a perfect metric. But I think it's a meaningful signal that is generally useful to consider. Eg. If Jon Skeet answers my C# question, I just have a strong prior that it's correct in a robust way.
It's an empirical question whether removing that signal would somehow improve the overall outcome (eg. removing that signal forces people use only use the heuristics that are based on the question itself instead of short circuiting that process in favor of an easier social signal).
But I would bet that the empirical test would show that including the reputation signal improves outcomes, on the general prior that more information is better than less information.
Edit: One thing that would change my bet: one feature of really high quality answers is that they are often long and made of parts, like they explain what to do, and what to avoid, and have details about why that is. You could track the length and detail of an answer as one of your heuristics for choosing a correct answer. If it turned out that people who have the correct answer were more likely to write answers with that sort of detail if their identity was hidden, AND it turned out that people use that sort of detail to decide if an answer were correct or not, then I could imagine better answers existing on the site overall, and those answers being more consistently upvoted.