My experience about 'culture fit' was that it was a code word for ageism. Sometimes they'd be even more direct if I pushed back. This wasn't at the end of a 45 minute interview, it was in the first five minutes after I'd already passed the technical review. Might not be true in every instance, but I absolutely cringe when I hear that phrase.
A candidate who is conflict-adverse, for instance, would not be a "culture fit" in an organization where decisions are made in screaming matches.
The problems come when your culture is toxic, particularly for certain types of people. If older candidates wouldn't be successful or happy in your work-20-hours-a-day-know-it-all culture, that's a culture problem, not a dog-whistle for ageism.
Some interviewers use "culture fit" to explain a not-like-me discrimination against people with a different gender, background, age, race, etc. Those people are generally toxic—but also indicative of an organizational culture that tolerates those biases by employees who are external-facing representatives of that company.
As a candidate, if you're ever rejected as a "bad culture fit", you probably dodged a bullet. That's a negative reflection on the company, not on you.