For all the rhetoric in the tech industry over diversity, "Culture Fit" seems a bit of a catch all exit hatch and anything but diverse.
The problem is that "Culture Fit" is often undefined - rarely is it explicit enough for us to say: "Our culture is x,y,z. This person showed this behavior so it's a no".
Although one thing I will say as an interviewer is I've had to reject candidates on "Culture Fit" in the past based on things like "This candidate talks way too much and it's so disruptive and frustrating that I literally can't get through the competency part of the interview". However, there is no way I'm going to feed that back to the candidate- so often Culture Fit is used as a neutral way of rejecting someone.