Very interesting. I've always thought that if I were to become good enough in a certain field to become "famous", then for all of my other passions, I would go anonymous. I've noticed there's a tendency people have to immediately disparage the work of someone who is really good at something if that work is not what they are well known for.
I'm not sure if there's a word or phrase for this phenomenon, but I'm pretty sure the tendency to disparage "polymath types" is a manifestation of envy and jealousy; how could one person possibly master two completely different things? Being great at one thing should be sufficient for anyone; being great at two is just unfair.
But the truth doesn't care about fairness. There are plenty of truths in the world that society is unwilling to acknowledge simply because of their implications (see Paul Graham's "What you can't say" for a fantastic exposition of this concept). Many believe that it's improbable for one person to be good at many different things (as though being "good at things" is independent for each different thing). I strongly disagree. I think that if someone has the tools or mindset to become very good at one thing, they are also very likely to be good at something else as well. That's why I don't like criticisms that are based on "well he's a physicist; he doesn't know about..." or "she's only an expert in neuroscience; who cares what she thinks about such-and-such".
People always want to include the background or history of a person in their analysis of that person's creations. Instead, I would rather all my work be evaluated purely on its own merit, without any extraneous knowledge biasing the reviewer.