Source matters. Always, for anything.
There is essentially no such thing as "inherent qualities" distinct from their source. Or rather, you can never actually know all the inherent qualities.
Knowledge of the sources history, character, motivations, and intentions is itself one of those inherent qualities.
If someone presents you with an idea, you often don't know if it's a good idea or just looks like a good idea because it was crafted to look good, in service of some unknown ultimate goal that you would not voluntarily choose to support if you knew everything.
Every scam in history is based on convincing a victim to harm themselves. Apparently, inherent qualities can be unknown, unknowable, or hidden. It's too late to contest that. It's already been happening for all of human history, and on every possible scale.
Judging something because the source has a known evil character is not wrong or counterproductive or unfair the way it is judging something because the source has pleasing charisma or lacks it.
If a Nazi says something, it's automatically not a wonderful idea in the first place, even if someone else later says the exact same thing and we all agree it's a wonderful idea then.
It's not wrong or illogical. It's just a slight of hand used by the dishonest (or the merely misguided) to try to make it seem wrong or hypocritical by pointing out only the surface similarity between that and prejudice based on nothing that stands up to scrutiny, when those are actually two very different things.
It's like a less obvious version of crying about being ddnied the freedom to deny others freedom.