I think the word "culture" is overloaded, such that it's really hard to talk about and you can make it mean whatever you want, but there's a use for it that I don't know a better way to describe in a word. Yes, there is "culture fit" in terms of things that don't matter, and where it does good to push a company outside of a comfort zone in order to get the best people and new perspectives.
But at a certain point, you need to get alignment between what work you find rewarding/fulfilling/exciting and what work a company tends to recognize and reward. This essay I think gets at the heart of it:
http://yosefk.com/blog/people-can-read-their-managers-mind.h...
If you are a programmer, which kind of organization do you think you would tend to want to work at, all things being equal: one where the organization's culture is set by people who come from programming backgrounds or sales backgrounds? It's not the same answer as if you were a salesperson. (And of course, there are programmers who probably prefer the culture at a place driven by salespeople, and vice versa. And you can start to think about why that might be.)
As an employee, what a good culture fit means is "someplace that values and rewards the things I want to be doing." As an employer, a good culture fit is someone who responds well to the incentives you have in place for them. Sometimes, cultures are just objectively bad; anywhere that judges programmers by how many lines of code they write alone is establishing perverse incentives. But sometimes cultures are not better or worse objectively, just different. Some people work better in a research culture and some people work better in a product culture, for instance. That's fine. But that's where finding out if you and an employee are a culture fit can benefit both parties, not just be an excuse to hire only people you like. It's not about beliefs, it's about what work is valued and how it's valued.