Also, Ellison was a great writer, but bringing attention to his work doesn't really advance the Cause.
The whole point of Invisible Man is that Ellison doesn't want to be reduced to a dehumanized representative of some collective. He is a person in his own right.
If you're an activist, the kind of thinking represented by Ellison is actively counterproductive. It's old-school Liberal stuff -- "outdated". It interferes with building the proper Consciousness.
Hence the objection to someone, and in particular a "white" person, working to advance Ellison without enough criticism. Instead you need someone who can "properly contextualize" Ellison, which is to say, to treat him with a little surface respect, but ultimately to make clear to readers that his work is somehow misguided. Nobody will go so far as to say "Uncle Tom" or "counter-revolutionary", but that's the subtext.
I wouldn't call this "talking like children", because the people are highly-educated adults and they understand exactly what they are doing.