I know what you mean about how genetics is often thrown up as the answer without much discussion on the complexities of human development.
The discussion on intelligence often suffers from this. There's a level where we are talking purely about the myelin sheath and conductance latencies among other speed factors in the brain (for example my brother-in-law is obsessed with learning about history, but he is "slow" despite his knowledge), but intelligence is driven by a desire to work through and explore the kinds of things (reading, writing, problem solving) that as a side effect give individuals the qualities that are judged as intelligent. And the presence of smart adults causing children to see these activities as exciting, and also pulling them up via active feedback and passive schaffolding, makes all the difference. Some people have introduced the notion of "multiple intelligence" to describe the fact that a difference focus of one's attention in developing as a person will lead to difference categories of competency.
And my reason for bringing this up is that some will try to explain intelligence in entirely genetic terms but this doesn't square with what is observed. Evolutionary psychology helps explain a lot of things, but it is all too easy to oversimplify.
If I were give my guess right now about how genetics influences traits, I would say that it obviously determines thousands of reflexive behaviors we perform, and of course it establishes potential competencies, but I think it additionally can set up guideposts that determine personality. It is known (if I remember correctly) that in some primitive organisms a single gene switches on avoidance versus more social behavior. There are plenty of ways in which even the complex behavior of behavior of humans could be guided by genetics into certain personality tendencies or strategies. For example if the cingulate cortex turns out to be a behavior-judging module in the brain among other things, then genetics could program the cingulate to fire a note of concern to the system, or fail to do the same, with regard to whatever complex decision is being made, whenever it involves a choice to be with people or not be with people.
Similarly if there are multiple genetically guided development strategies for handling social information, and some of them just don't scale that well to larger groups of people, then perhaps this would produce anxiety and a sense of not being able to handle the situation, among people who have the more brittle genetic strategy. But the same argument could be made for voluntary control that would influence the underlying cognitive strategy, because not enough is known yet.