In my experience "Candidate X isn't a good cultural fit" has almost always been used as short had for either "I don't want us to hire them, but I can't articulate why in way I am comfortable with" or less often "What an asshole". I don't think I've ever been part of a conversation about "cultural fit" that wasn't swimming in bullshit, or at least wading in it.
In almost every case, they've ended up later either demonstrating some unprofessional behavior, like having an abusive temper, or been revealed to be incompetent in some aspect of the job. And yet, I still wouldn't feel comfortable turning somebody down purely based on a gut feeling. It's made me pretty uncomfortable about participating in hiring process at all.