I think you're spot on with the "they think you're too uptight" hypothesis. Small teams can be really anal when it comes to culture fit, for better or for worse, and unfortunately when it comes to this people can be really prejudiced and ofter rely on more "superficial" factors such as gender and age. I personally work at a startup with a lot of pretty brilliant people, but that has a pretty bro-ey feel at times. I p…
I've had to turn down several otherwise-qualified candidates recently for culture fit reasons. While it would be awesome if all qualified candidates were created equal, their technical qualifications are only one piece of the puzzle.
In reality, the culture fit and the ability to work with the rest of the team is key in a small company. Everybody has to deal with everybody, so if anybody has issues dealing with anybody you can run into some major friction that'll cost you more than you gain.
At the end of the day, the way to look at it isn't "they're qualified but they don't fit the culture", but rather "they're not qualified because they don't fit the culture". It goes a long way toward eliminating the mixed feelings.