Critical Race Theory emerged from mid-20th century Black Power and Colonial Independence movements, which often either were literally Marxist movements in their own right, or intimately tied to them. There are countless examples of the close relationship, but one of the most pertinent embodiments would in be Frantz Fanon, who is one of the most important if not
the most important ideological fathers of both Critical Race Theory (as well as other, older movements) and to a lesser extent several Marxist movements in post-colonial Africa.
A lot of anti-CRT rhetoric is clearly trying to use the Marxism link to discredit, out-of-hand, CRT. American politics has long been like that--guilt by association with Socialism. I understand the reflexive instinct to push back. But if you try to deny that the links exist, all you're doing is discrediting yourself and others, bolstering those pundits' credibility. The links absolutely exist and are substantial. The nuance is that they're not substantial in the ways that their rhetoric implies. (It's like if you took Christianity and replaced Jesus with Santa Claus. You couldn't plausibly claim the new religion wasn't an offshoot of Christianity, and much of it would be indecipherable without understanding historical Christianity and the figure of Jesus. But it would simultaneously be true that the new religion would in some of its most fundamental dimensions be entirely incompatible and incomparable to Christianity.)
If anybody wants to have an informed opinion on these debates, they could do much worse than reading Frantz Fanon. IMO, CRT begins and ends with Frantz Fanon. If you read one of his seminal works, "Black Skin, White Faces", you've read all you'll ever need to read regarding the ideological underpinnings of not only CRT but much of modern liberal American identity politics, either because it descends directly from Fanon, or because Fanon expertly articulates the experiences and reasoning. I found it a very powerful and enlightening book.[1] But Fanon takes (in the book, but especially in subsequent works and the arc of his entire public life) the ideology to its logical conclusion, which is permanent apartheid, and therefore lays bare the stark choice being made when one adopts the axioms shared between liberal identity politics and Fanon's philosophy. I can't refute Fanon's ultimate conclusion that black-skinned minorities will never be able to find equality among a white-skinned majority (or, presumably, vice-versa if the historical script were flipped), but I choose to believe that there must exist some way to get there; otherwise as Fanon persuasively argues, all identity politics can possibly do is mitigate some of the worst injustices, but ultimately never eradicate the fundamental inequities. And therein lies much of Fanon's motivation for his Marxism advocacy--he thought the only way forward was for black countries to go their own way, and that necessarily involved adopting Marxism as capitalist systems would ultimately benefit the white, colonial establishment at the expense of black countries and communities.
Also, FWIW, modern Marxist scholarship provides some of the most vehement and technical opposition to CRT for precisely the reason you hinted at--replacing class conflict with racial conflict is anathema to Marxism. But when you read that scholarship, it's also obvious that CRT and Marxism use nearly identical language. CRT is indisputably descended from the culture of Marxist scholarship. That's also why modern Marxists are so adept at counterarguments--they're familiar with many of the underlying modes of reasoning. (20th century Marxists weren't very interested in opposing these philosophies for geopolitical reasons.)
[1] I first read it 20 years ago, before CRT entered mainstream liberal discourse, and before I had any knowledge of CRT as a school of thought or movement. When I did learn of CRT, the parallels to Fanon's ideas were immediately obvious. Until then mainstream liberal identity politics hadn't moved much beyond, say, Cornel West's philosophy in "Race Matters". West has always been very careful to not cross the line into race essentializing territory, and it's for that reason that in the past several years West has been condemned by the identity politics movement as it continues to follow identity politics to its logical end.